Oct 30 21:00:48.529598 ip-10-0-140-12 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:00:48.966266 ip-10-0-140-12 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:48.966266 ip-10-0-140-12 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:00:48.966266 ip-10-0-140-12 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:48.966266 ip-10-0-140-12 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:48.969954 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969845    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:48.969954 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969954    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969959    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969963    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969966    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969969    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969973    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969975    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969978    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969981    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969984    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969986    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969989    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969991    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969994    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969996    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.969999    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970002    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970005    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970007    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970016    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:48.970028 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970019    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970021    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970024    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970027    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970030    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970033    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970036    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970039    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970041    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970044    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970046    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970049    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970052    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970054    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970057    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970059    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970062    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970065    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970067    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:48.970536 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970070    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970073    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970075    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970078    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970081    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970083    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970086    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970088    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970092    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970096    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970099    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970101    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970104    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970106    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970109    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970112    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970115    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970118    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970121    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:48.971034 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970123    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970126    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970128    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970131    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970133    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970136    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970138    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970141    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970144    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970146    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970149    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970152    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970155    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970159    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970161    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970164    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970166    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970169    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970172    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970174    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:48.971528 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970177    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970179    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970182    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970184    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970187    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970190    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970193    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970195    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970199    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970202    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970204    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970207    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970209    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970212    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970215    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970217    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970220    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970222    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970224    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970227    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:48.972048 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970229    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970232    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970234    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970237    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970239    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970244    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.970249    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971020    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971027    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971030    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971033    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971036    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971040    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971043    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971046    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971049    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971052    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971055    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971057    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:48.972556 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971060    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971063    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971066    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971068    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971071    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971074    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971077    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971081    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971085    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971087    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971090    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971093    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971095    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971098    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971100    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971103    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971105    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971108    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971110    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:48.973193 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971113    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971116    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971119    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971121    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971124    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971127    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971130    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971133    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971135    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971138    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971140    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971143    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971146    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971148    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971151    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971153    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971156    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971158    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971161    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971163    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:48.973710 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971166    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971169    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971171    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971174    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971177    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971179    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971182    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971184    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971187    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971189    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971193    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971197    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971200    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971203    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971206    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971209    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971211    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971214    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971217    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:48.974238 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971220    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971223    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971225    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971228    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971231    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971234    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971236    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971239    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971242    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971244    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971247    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971249    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971252    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971255    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971257    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971260    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971262    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971265    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971268    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971271    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:48.974740 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971273    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971276    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971279    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971282    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971286    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971288    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971291    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971294    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971297    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971299    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971302    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971305    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971307    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971310    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971327    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971330    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971804    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971809    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971812    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971818    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:48.975242 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971822    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971824    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971827    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971830    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971832    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971835    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971838    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971841    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971843    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971846    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971852    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971855    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971858    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971860    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971863    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971866    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971868    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971871    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971874    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971876    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:48.975767 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971880    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971882    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971885    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971889    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971893    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971896    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971898    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971901    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971903    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971906    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971908    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971913    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971915    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971918    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971921    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971923    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971927    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971930    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:48.976276 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971934    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971937    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971940    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971943    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971947    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971949    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971952    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971954    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971957    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971960    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971962    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971965    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971967    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971970    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971972    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971981    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971984    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971986    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971989    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971992    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:48.976757 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971994    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971997    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.971999    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972002    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972004    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972011    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972014    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972017    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972020    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972022    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972025    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972028    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972030    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972033    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972036    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972038    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972042    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972045    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972048    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:48.977255 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972050    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972053    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972056    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972059    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972062    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972064    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972067    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972070    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972072    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972075    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972078    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972081    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972083    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972086    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972088    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972091    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972093    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972096    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972099    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972103    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:48.977758 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972105    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972108    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972110    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972113    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.972115    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973226    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973237    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973243    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973248    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973253    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973256    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973261    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973265    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973268    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973271    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973275    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973278    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973281    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973284    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973287    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973290    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973293    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973296    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973301    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:00:48.978246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973304    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973308    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973311    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973325    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973330    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973333    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973337    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973347    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973350    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973354    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973357    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973360    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973363    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973367    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973370    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973373    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973376    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973380    2574 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973383    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973387    2574 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973390    2574 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973393    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973396    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973399    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:00:48.978875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973403    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973406    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973410    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973413    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973421    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973424    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973427    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973431    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973434    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973437    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973440    2574 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973444    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973448    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973451    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973455    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973458    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973461    2574 flags.go:64] FLAG: --help="false"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973465    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973468    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973472    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973474    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973478    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973481    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:00:48.979535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973484    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973487    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973491    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973493    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973496    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973500    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973502    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973505    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973508    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973511    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973514    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973517    2574 flags.go:64] FLAG: --lock-file=""
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973520    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973523    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973526    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973531    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973534    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973538    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973541    2574 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973544    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973547    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973550    2574 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973553    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973558    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973561    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:00:48.980116 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973565    2574 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973568    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973571    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973574    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973576    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973579    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973582    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973585    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973592    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973596    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973599    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973602    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973605    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973607    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973610    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973613    2574 flags.go:64] FLAG: --port="10250"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973616    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973619    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0859b3f8478b12bef"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973622    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973625    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973628    2574 flags.go:64] FLAG: --register-node="true"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973631    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973635    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973637    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973641    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973644    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:00:48.981104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973648    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973651    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973654    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973657    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973661    2574 flags.go:64] FLAG: --runonce="false"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973664    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973667    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973671    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973673    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973676    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973679    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973682    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973687    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973690    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973693    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973696    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973699    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973703    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973706    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973708    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973714    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973717    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973720    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973724    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973727    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:00:48.982389 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973730    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973733    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973737    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973740    2574 flags.go:64] FLAG: --v="2"
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973744    2574 flags.go:64] FLAG: --version="false"
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973748    2574 flags.go:64] FLAG: --vmodule=""
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973752    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.973756    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973892    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973895    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973898    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973902    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973905    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973908    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973910    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973913    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973915    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973918    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973920    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973924    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973927    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:48.983488 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973930    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973932    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973935    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973937    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973940    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973942    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973945    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973947    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973950    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973953    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973955    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973958    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973960    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973963    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973965    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973968    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973970    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973973    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973976    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973979    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:48.984530 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973981    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973985    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973990    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973993    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973996    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.973999    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974002    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974005    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974007    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974009    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974013    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974016    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974018    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974021    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974024    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974026    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974029    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974031    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974034    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:48.985447 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974036    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974039    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974042    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974044    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974047    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974049    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974052    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974054    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974057    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974059    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974062    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974064    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974067    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974070    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974072    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974076    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974079    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974081    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974084    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974087    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:48.986111 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974089    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974092    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974094    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974098    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974100    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974103    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974105    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974108    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974112    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974115    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974118    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974121    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974124    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974126    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974128    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974131    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974133    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974136    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974138    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:48.986808 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974141    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974143    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974146    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974149    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974151    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974154    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974156    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974159    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974163    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974166    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974169    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974171    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974174    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974176    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974179    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.974206    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:48.987631 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974243    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974249    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974253    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974255    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974258    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974261    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974263    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974266    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974268    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974271    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974274    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974276    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974279    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974281    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974284    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974287    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974289    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974292    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974295    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974299    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:48.988230 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974302    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974305    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974308    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974310    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974334    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974337    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974340    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974342    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974345    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974348    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974351    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974353    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974356    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974361    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974363    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974366    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974369    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974371    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974374    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:48.988782 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974377    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974379    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974382    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974384    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974387    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974389    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974392    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974395    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974397    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974400    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974403    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974405    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974408    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974410    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974413    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974416    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974418    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974423    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974426    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974429    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:48.989570 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974431    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974434    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974436    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974439    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974441    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974444    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974450    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974452    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974455    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974457    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974460    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974463    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974465    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974468    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974470    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974473    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974476    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974479    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974481    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:48.990301 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974485    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974489    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974492    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974495    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974499    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974501    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974504    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974506    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974509    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974511    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974515    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974517    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974520    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974523    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974526    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974528    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974531    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974534    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974536    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974540    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:48.991074 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974543    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974545    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974548    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974550    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974553    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974555    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974558    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.974561    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.983332    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.983354    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.983376    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.983383    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983492    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983500    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983505    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983509    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983513    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983517    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983521    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:48.991608 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983526    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983530    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983535    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983539    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983543    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983548    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983552    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983556    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983561    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983565    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983569    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983574    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983579    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983583    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983588    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983593    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983600    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983606    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983612    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:48.992089 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983616    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983621    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983626    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983641    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983647    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983652    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983657    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983661    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983666    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983670    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983675    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983679    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983683    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983688    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983692    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983697    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983701    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983705    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983710    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:48.992616 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983714    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983719    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983723    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983727    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983731    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983736    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983740    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983744    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983749    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983754    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983758    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983762    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983766    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983771    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983776    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983779    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983784    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983797    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983802    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:48.993144 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983806    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983811    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983815    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983819    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983823    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983827    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983832    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983836    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983840    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983844    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983848    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983853    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983857    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983862    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983866    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983870    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983874    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983879    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983883    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983888    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:48.993689 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983892    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983897    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983901    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983905    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983909    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983913    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983918    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983931    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983936    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983941    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983945    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983958    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983963    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983967    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983972    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983976    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983980    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983984    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983989    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983993    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:48.994208 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.983997    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984001    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.984046    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984390    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984402    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984407    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984412    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984416    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984421    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984425    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984429    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984434    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984438    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984443    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984447    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984452    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:48.994750 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984456    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984460    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984465    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984469    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984474    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984478    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984482    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984486    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984499    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984505    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984509    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984514    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984518    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984522    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984526    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984530    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984534    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984538    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984542    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984546    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:48.995206 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984550    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984554    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984558    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984562    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984569    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984575    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984580    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984585    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984589    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984593    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984598    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984602    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984606    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984611    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984616    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984620    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984624    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984628    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984635    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:48.995763 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984642    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984647    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984662    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984667    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984671    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984675    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984680    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984684    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984688    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984693    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984697    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984701    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984705    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984710    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984714    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984718    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984722    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984726    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984731    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984735    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:48.996236 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984739    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984744    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984748    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984752    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984756    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984761    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984765    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984769    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984774    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984779    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984783    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984787    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984792    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984796    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984800    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984812    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984816    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984821    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984825    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984829    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:48.996759 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984834    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984838    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984842    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984846    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984850    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984854    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984858    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984862    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984866    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984870    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984875    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984879    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984883    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984887    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.984932    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:48.997298 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.984995    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985002    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985007    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985012    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985016    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985020    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985024    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985029    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985033    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985038    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985042    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985046    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985050    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985064    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985068    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985072    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985077    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985082    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985086    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985090    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:48.997719 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985094    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985099    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985103    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985108    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985112    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985116    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985120    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985125    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985129    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985133    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985137    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985143    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985150    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985155    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985160    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985165    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985169    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985173    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985178    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985182    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:48.998224 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985186    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985190    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985196    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985200    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985205    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985209    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985223    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985228    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985233    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985237    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985241    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985245    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985249    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985254    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985258    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985262    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985266    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985270    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985274    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:48.998860 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985279    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985283    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985287    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985292    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985298    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985302    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985306    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985311    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985335    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985345    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985349    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985353    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985357    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985362    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985366    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985370    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985375    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985379    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985383    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985387    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:48.999338 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985399    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985404    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985408    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985412    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985417    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985421    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985425    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985430    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985434    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985438    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985442    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985446    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985450    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985454    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985458    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985462    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985467    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985472    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985475    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:48.999849 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985480    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985484    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985489    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985500    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985505    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985509    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985513    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:48.985517    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.985613    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.988835    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.990687    2574 server.go:1014] "Starting client certificate rotation"
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.990795    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.990835    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.993014    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:00:49.000517 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:48.994153    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:00:49.002261 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.002244    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:00:49.006685 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.006668    2574 log.go:25] "Validated CRI v1 image API"
Oct 30 21:00:49.007513 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.007491    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:00:49.013019 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.013000    2574 fs.go:135] Filesystem UUIDs: map[79103255-552b-4ff4-8a93-44b1a133ad37:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7f0ebb78-9780-40a9-9652-bb727666486d:/dev/nvme0n1p3]
Oct 30 21:00:49.013095 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.013018    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:00:49.019251 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.019227    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:00:49.019569 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.019453    2574 manager.go:217] Machine: {Timestamp:2026-10-30 21:00:49.017918802 +0000 UTC m=+0.379456899 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083701 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29362e93c2570897bfad7ccf1e731a SystemUUID:ec29362e-93c2-5708-97bf-ad7ccf1e731a BootID:a890f721-113a-4c8b-aac7-d1a347654935 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:c8:20:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:c8:20:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:6a:09:36:2c:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:00:49.019569 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.019568    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:00:49.019704 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.019693    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:00:49.021074 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.021052    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:00:49.021274 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.021076    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-12.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:00:49.021336 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.021277    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:00:49.021336 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.021285    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:00:49.021336 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.021312    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:00:49.021434 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.021350    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:00:49.022644 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.022633    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:00:49.027355 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.027342    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:00:49.027485 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.027476    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:00:49.027515 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.027509    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:00:49.027545 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.027519    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:00:49.027545 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.027529    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:00:49.027602 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.027551    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:00:49.027602 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.027559    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:00:49.027966 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.027954    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:00:49.032007 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.031930    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:00:49.037568 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.037550    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wnjq"
Oct 30 21:00:49.039686 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.039669    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:00:49.040550 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.040538    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:00:49.040587 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.040569    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:00:49.040724 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.040714    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:00:49.040766 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.040730    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:00:49.040766 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.040736    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:00:49.040766 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.040743    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:00:49.040766 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.040749    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:00:49.040766 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.040754    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:00:49.041439 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.041429    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:00:49.041470 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.041440    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:00:49.041470 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.041448    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:00:49.041470 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.041453    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:00:49.041470 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.041462    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:00:49.041470 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.041471    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:00:49.042804 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.042794    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:00:49.042804 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.042805    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:00:49.043262 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.043245    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wnjq"
Oct 30 21:00:49.046878 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.046863    2574 server.go:1263] "Started kubelet"
Oct 30 21:00:49.046987 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.046946    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:00:49.047040 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.047008    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:00:49.047194 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.047160    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:00:49.047697 ip-10-0-140-12 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:00:49.049128 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.049043    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-12.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:00:49.049213 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.049186    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:00:49.049264 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.049188    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-12.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:00:49.051537 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.051521    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:00:49.052778 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.052755    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:00:49.053722 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.053701    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:00:49.058778 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.058763    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:00:49.059344 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.059310    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:00:49.059599 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.059566    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.060093 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060077    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:00:49.060093 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060079    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:00:49.060236 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060124    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:00:49.060425 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060345    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:00:49.060516 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060432    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:00:49.060677 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060661    2574 factory.go:153] Registering CRI-O factory
Oct 30 21:00:49.060749 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060682    2574 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:00:49.060749 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060732    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:00:49.060749 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060741    2574 factory.go:55] Registering systemd factory
Oct 30 21:00:49.060749 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060750    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:00:49.060928 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060765    2574 factory.go:103] Registering Raw factory
Oct 30 21:00:49.060928 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.060774    2574 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:00:49.061902 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.061882    2574 manager.go:319] Starting recovery of all containers
Oct 30 21:00:49.062841 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.062815    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:49.065946 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.065925    2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-12.us-east-2.compute.internal\" not found" node="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.072543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.072516    2574 manager.go:324] Recovery completed
Oct 30 21:00:49.077391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.077377    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:49.079482 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.079466    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:49.079543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.079498    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:49.079543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.079509    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:49.080037 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.080023    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:00:49.080092 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.080039    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:00:49.080092 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.080059    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:00:49.080166 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.080028    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:49.080875 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.080860    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:49.080927 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.080887    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:49.080927 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.080896    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:49.082403 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.082393    2574 policy_none.go:50] "Start"
Oct 30 21:00:49.082446 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.082410    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:00:49.082446 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.082421    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:00:49.084543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.084531    2574 policy_none.go:44] "Start"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.132916    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.137616    2574 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.137645    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.137659    2574 server.go:87] "Starting device plugin registration server"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.137890    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.137901    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.138013    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.138083    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.138089    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.138577    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.138625    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.139092    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.140461    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.140484    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.140507    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:00:49.143672 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.140544    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:00:49.148751 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.148736    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:49.238748 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.238718    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:49.239830 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.239812    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:49.239952 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.239846    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:49.239952 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.239865    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:49.239952 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.239899    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.240939 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.240907    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"]
Oct 30 21:00:49.241004 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.240980    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:49.241756 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.241740    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:49.241829 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.241770    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:49.241829 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.241781    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:49.243902 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.243889    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:49.244031 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.244018    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.244089 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.244046    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:49.244622 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.244593    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:49.244622 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.244610    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:49.244622 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.244622    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:49.244816 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.244633    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:49.244816 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.244637    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:49.244816 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.244643    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:49.246806 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.246790    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.246895 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.246822    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:49.247559 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.247543    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:49.247635 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.247566    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:49.247635 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.247582    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:49.249143 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.249123    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.249143 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.249142    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-12.us-east-2.compute.internal\": node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.260616 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.260589    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.260729 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.260660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.260729 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.260684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.260729 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.260706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/070c8221ace6d9936bb76d676bde1ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"070c8221ace6d9936bb76d676bde1ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.273255 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.273230    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found" node="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.279707 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.279689    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found" node="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.361249 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.361222    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.361249 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.361244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.361472 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.361271    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/070c8221ace6d9936bb76d676bde1ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"070c8221ace6d9936bb76d676bde1ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.361472 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.361287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.361472 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.361329    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.361472 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.361357    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.361472 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.361372    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/070c8221ace6d9936bb76d676bde1ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"070c8221ace6d9936bb76d676bde1ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.462301 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.462271    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.562963 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.562897    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.578113 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.578088    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.585185 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.585169    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:49.663736 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.663705    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.764091 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.764054    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.864500 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.864430    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.964900 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:49.964871    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:49.990065 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.990037    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:00:49.990661 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.990182    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:00:49.990661 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:49.990222    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:00:50.049646 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.049586    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:55:50 +0000 UTC" deadline="2028-05-18 20:54:10.388134271 +0000 UTC"
Oct 30 21:00:50.049646 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.049640    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13583h53m20.338497142s"
Oct 30 21:00:50.064938 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:50.064917    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:50.165774 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:50.165744    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:50.243413 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:50.243362    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070c8221ace6d9936bb76d676bde1ea5.slice/crio-22907cb2e597dcd95cf4966c60ca85160f6ed2105a67483719654b28242c9c39 WatchSource:0}: Error finding container 22907cb2e597dcd95cf4966c60ca85160f6ed2105a67483719654b28242c9c39: Status 404 returned error can't find the container with id 22907cb2e597dcd95cf4966c60ca85160f6ed2105a67483719654b28242c9c39
Oct 30 21:00:50.243754 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:50.243732    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57946f0cf74e97be779f3304bd4cc78.slice/crio-e316f7f815709ff003f17b061fdc4173c5d92d2ad392678fb90e22e8291f04b7 WatchSource:0}: Error finding container e316f7f815709ff003f17b061fdc4173c5d92d2ad392678fb90e22e8291f04b7: Status 404 returned error can't find the container with id e316f7f815709ff003f17b061fdc4173c5d92d2ad392678fb90e22e8291f04b7
Oct 30 21:00:50.251278 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.251259    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:00:50.266200 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:50.266182    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:50.366628 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:50.366590    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:50.466999 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:50.466927    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-12.us-east-2.compute.internal\" not found"
Oct 30 21:00:50.484639 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.484614    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:50.559440 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.559414    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:50.560371 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.560355    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:50.571472 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.571453    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:00:50.572560 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.572548    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:00:50.582765 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:50.582743    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:00:52.959223 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.959193    2574 apiserver.go:52] "Watching apiserver"
Oct 30 21:00:52.968202 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.968176    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:00:52.972976 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.972939    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv","openshift-image-registry/node-ca-wrffd","openshift-multus/multus-5sxfl","openshift-network-diagnostics/network-check-target-kwb4j","openshift-multus/multus-additional-cni-plugins-mfwjh","openshift-network-operator/iptables-alerter-vmp52","kube-system/global-pull-secret-syncer-lbvj4","openshift-ovn-kubernetes/ovnkube-node-4cz8g","openshift-cluster-node-tuning-operator/tuned-9hs5s","openshift-dns/node-resolver-r7gpg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-gbg4d","kube-system/konnectivity-agent-r2kvg"]
Oct 30 21:00:52.975526 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.975505    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:52.978033 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.978000    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:52.978134 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:52.978092    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:00:52.982121 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.982098    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:00:52.982244 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.982230    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qbtrh\""
Oct 30 21:00:52.982305 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.982248    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:00:52.982305 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.982293    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:00:52.982411 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.982335    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:52.983100 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.983077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:52.984975 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.984956    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:52.985081 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:52.985018    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:00:52.988319 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.988058    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:52.989813 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.989794    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:00:52.990651 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.990634    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:52.993049 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.993032    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:00:52.995721 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.995701    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:52.999180 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:52.998940    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000359    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-k8s-cni-cncf-io\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000423    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-socket-dir-parent\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-kubelet\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000478    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-node-log\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000497    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93767113-7f48-4ed1-bb24-d6caaa025dfb-konnectivity-ca\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6dv\" (UniqueName: \"kubernetes.io/projected/595d39f4-1364-4e9e-87ae-09120d783bdd-kube-api-access-wn6dv\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-system-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-multus\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000564    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxws\" (UniqueName: \"kubernetes.io/projected/0a6e387b-47f4-4966-b197-e2e73150d514-kube-api-access-ntxws\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000578    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-config\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67clk\" (UniqueName: \"kubernetes.io/projected/e0130da1-910e-4623-834b-40b494b2309d-kube-api-access-67clk\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-conf-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000658    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-var-lib-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.000834 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-ovn\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93767113-7f48-4ed1-bb24-d6caaa025dfb-agent-certs\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-etc-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000933    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-multus-daemon-config\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.000971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-multus-certs\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gth79\" (UniqueName: \"kubernetes.io/projected/66f1f33b-3e12-4a49-9ea7-918c9bf78265-kube-api-access-gth79\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001053    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-netd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f71f208a-7acb-47f5-b6c5-eb53347e18f9-host-slash\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001139    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f71f208a-7acb-47f5-b6c5-eb53347e18f9-iptables-alerter-script\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001184    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/595d39f4-1364-4e9e-87ae-09120d783bdd-host\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001212    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-cnibin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-kubelet\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-hostroot\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001333    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-systemd-units\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001358    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-systemd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.001572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001388    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-env-overrides\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk79d\" (UniqueName: \"kubernetes.io/projected/f71f208a-7acb-47f5-b6c5-eb53347e18f9-kube-api-access-mk79d\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001510    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/595d39f4-1364-4e9e-87ae-09120d783bdd-serviceca\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001537    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-cni-binary-copy\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001607    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-netns\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001631    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-system-cni-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001652    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xxh\" (UniqueName: \"kubernetes.io/projected/0ba46ac0-612c-4964-9454-8ae5e42bd728-kube-api-access-d8xxh\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001688    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-bin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-netns\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-log-socket\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0130da1-910e-4623-834b-40b494b2309d-ovn-node-metrics-cert\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001890    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-script-lib\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.002222 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-etc-kubernetes\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.002878 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.001983    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.002878 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.002037    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-bin\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.002878 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.002065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:53.002878 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.002102    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cnibin\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.002878 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.002128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.002878 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.002151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-slash\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.003832 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.003758    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.003889 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.003822    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:00:53.006510 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.006488    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:00:53.024524 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.024501    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l87dh"
Oct 30 21:00:53.035930 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.035892    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l87dh"
Oct 30 21:00:53.077489 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.077436    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"c57946f0cf74e97be779f3304bd4cc78","Type":"ContainerStarted","Data":"e316f7f815709ff003f17b061fdc4173c5d92d2ad392678fb90e22e8291f04b7"}
Oct 30 21:00:53.078791 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.078758    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"070c8221ace6d9936bb76d676bde1ea5","Type":"ContainerStarted","Data":"22907cb2e597dcd95cf4966c60ca85160f6ed2105a67483719654b28242c9c39"}
Oct 30 21:00:53.080119 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.080093    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:00:53.080232 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.080142    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:00:53.080232 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.080183    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:00:53.080342 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.080237    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:00:53.080397 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.080350    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:00:53.080397 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.080354    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:00:53.081554 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.081461    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:00:53.081554 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.081533    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:00:53.081554 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.081535    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:00:53.081762 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.081540    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082061    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082125    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082162    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082178    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082192    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfb4q\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082205    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v7745\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082194    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082225    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082275    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082294    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wx578\""
Oct 30 21:00:53.082327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082301    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g68zh\""
Oct 30 21:00:53.082937 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.082342    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:00:53.083816 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.083797    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mwc48\""
Oct 30 21:00:53.083904 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.083862    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hspc2\""
Oct 30 21:00:53.083904 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.083885    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:00:53.083904 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.083894    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:00:53.084045 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.083912    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:00:53.084045 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.083941    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:00:53.084045 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.083805    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-99mk5\""
Oct 30 21:00:53.084045 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.084025    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x8hl8\""
Oct 30 21:00:53.084045 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.084034    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:00:53.084249 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.084198    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:00:53.091708 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.091683    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:00:53.103336 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.103482 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103342    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-slash\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.103482 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-kubernetes\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.103482 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103391    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-var-lib-kubelet\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.103482 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-k8s-cni-cncf-io\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.103482 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103485    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-slash\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103508    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-k8s-cni-cncf-io\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103548    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103558    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-lib-modules\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-socket-dir-parent\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103644    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-kubelet\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-node-log\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-modprobe-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.103754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-tuned\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-tmp\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103776    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-kubelet\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-socket-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103888    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93767113-7f48-4ed1-bb24-d6caaa025dfb-konnectivity-ca\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6dv\" (UniqueName: \"kubernetes.io/projected/595d39f4-1364-4e9e-87ae-09120d783bdd-kube-api-access-wn6dv\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-system-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103951    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-socket-dir-parent\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.103971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-multus\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104027    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxws\" (UniqueName: \"kubernetes.io/projected/0a6e387b-47f4-4966-b197-e2e73150d514-kube-api-access-ntxws\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.104087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-config\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.104371 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67clk\" (UniqueName: \"kubernetes.io/projected/e0130da1-910e-4623-834b-40b494b2309d-kube-api-access-67clk\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.104371 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-conf\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.104371 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104176    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-conf-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.104371 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104200    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.104371 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104203    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2492dc3b-9893-45f5-af15-314312a0a18f-hosts-file\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.104371 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104279    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.104371 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-var-lib-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.104594 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104381    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-node-log\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.104594 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104393    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2492dc3b-9893-45f5-af15-314312a0a18f-tmp-dir\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.104594 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-ovn\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.104594 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104474    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-conf-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.104594 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104489    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-system-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.104594 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104506    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-var-lib-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.104594 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104549    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-multus\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.104594 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-kubelet-config\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.104968 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104911    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-ovn\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.104968 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104943    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93767113-7f48-4ed1-bb24-d6caaa025dfb-agent-certs\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:00:53.104968 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-etc-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.105108 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.105108 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.104992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-sys\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.105108 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-multus-daemon-config\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.105108 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105038    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93767113-7f48-4ed1-bb24-d6caaa025dfb-konnectivity-ca\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:00:53.105108 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-multus-certs\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.105108 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gth79\" (UniqueName: \"kubernetes.io/projected/66f1f33b-3e12-4a49-9ea7-918c9bf78265-kube-api-access-gth79\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.105369 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105264    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-etc-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.105369 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105297    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.105369 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105315    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:00:53.105369 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105347    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-multus-certs\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.105775 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-netd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.105888 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105826    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.105888 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105857    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-multus-daemon-config\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.105888 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105873    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f71f208a-7acb-47f5-b6c5-eb53347e18f9-host-slash\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.105977 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105924    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.105977 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105928    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f71f208a-7acb-47f5-b6c5-eb53347e18f9-host-slash\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.105977 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105924    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-config\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.105977 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7zft\" (UniqueName: \"kubernetes.io/projected/7e65d887-3e39-400b-8a4f-7953a5a680e9-kube-api-access-c7zft\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.105977 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105965    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-netd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.106160 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.105997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f71f208a-7acb-47f5-b6c5-eb53347e18f9-iptables-alerter-script\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.106160 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/595d39f4-1364-4e9e-87ae-09120d783bdd-host\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.106160 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-cnibin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106160 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106108    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/595d39f4-1364-4e9e-87ae-09120d783bdd-host\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.106160 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-kubelet\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-hostroot\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106213    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-kubelet\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106223    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-cnibin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106287    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-hostroot\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-systemd-units\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-systemd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-env-overrides\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk79d\" (UniqueName: \"kubernetes.io/projected/f71f208a-7acb-47f5-b6c5-eb53347e18f9-kube-api-access-mk79d\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:53.106391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106393    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/595d39f4-1364-4e9e-87ae-09120d783bdd-serviceca\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106424    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysconfig\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-systemd\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-host\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106466    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-registration-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106519    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-device-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bp5\" (UniqueName: \"kubernetes.io/projected/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kube-api-access-59bp5\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106564    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-cni-binary-copy\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-netns\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-system-cni-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xxh\" (UniqueName: \"kubernetes.io/projected/0ba46ac0-612c-4964-9454-8ae5e42bd728-kube-api-access-d8xxh\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106642    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.106831 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106649    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f71f208a-7acb-47f5-b6c5-eb53347e18f9-iptables-alerter-script\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-systemd-units\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106726    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-systemd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106729    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-bin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-bin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-netns\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-log-socket\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0130da1-910e-4623-834b-40b494b2309d-ovn-node-metrics-cert\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-script-lib\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mqhx\" (UniqueName: \"kubernetes.io/projected/2492dc3b-9893-45f5-af15-314312a0a18f-kube-api-access-8mqhx\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-etc-kubernetes\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106898    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-bin\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-run\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.106997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-sys-fs\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-dbus\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.107298 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107028    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-env-overrides\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107048    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cnibin\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107134    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cnibin\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107144    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-etc-kubernetes\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107183    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-log-socket\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.107287    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107322    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-netns\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-bin\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.107378    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:53.60734621 +0000 UTC m=+3.038532153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:53.107783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107759    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/595d39f4-1364-4e9e-87ae-09120d783bdd-serviceca\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.108215 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.107287    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.108318 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.108243    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-netns\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.108395 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.108340    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.108395 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.108381    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-system-cni-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.109701 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.109608    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-cni-binary-copy\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.109701 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.109608    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-script-lib\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.109849 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.109789    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.110088 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.110024    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93767113-7f48-4ed1-bb24-d6caaa025dfb-agent-certs\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:00:53.110604 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.110583    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0130da1-910e-4623-834b-40b494b2309d-ovn-node-metrics-cert\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.114903 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.114881    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxws\" (UniqueName: \"kubernetes.io/projected/0a6e387b-47f4-4966-b197-e2e73150d514-kube-api-access-ntxws\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.115061 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.115045    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:53.115139 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.115071    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:53.115139 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.115084    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5c8mg for pod openshift-network-diagnostics/network-check-target-kwb4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:53.115244 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.115174    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg podName:a6c067ea-93f5-4019-b9e4-100a1c8cb53c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:53.615154961 +0000 UTC m=+3.046340901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5c8mg" (UniqueName: "kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg") pod "network-check-target-kwb4j" (UID: "a6c067ea-93f5-4019-b9e4-100a1c8cb53c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:53.115787 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.115763    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gth79\" (UniqueName: \"kubernetes.io/projected/66f1f33b-3e12-4a49-9ea7-918c9bf78265-kube-api-access-gth79\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.117391 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.117343    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67clk\" (UniqueName: \"kubernetes.io/projected/e0130da1-910e-4623-834b-40b494b2309d-kube-api-access-67clk\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.119538 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.119515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6dv\" (UniqueName: \"kubernetes.io/projected/595d39f4-1364-4e9e-87ae-09120d783bdd-kube-api-access-wn6dv\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.125054 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.123592    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk79d\" (UniqueName: \"kubernetes.io/projected/f71f208a-7acb-47f5-b6c5-eb53347e18f9-kube-api-access-mk79d\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.125777 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.125751    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xxh\" (UniqueName: \"kubernetes.io/projected/0ba46ac0-612c-4964-9454-8ae5e42bd728-kube-api-access-d8xxh\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:53.208373 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208336    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mqhx\" (UniqueName: \"kubernetes.io/projected/2492dc3b-9893-45f5-af15-314312a0a18f-kube-api-access-8mqhx\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.208541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-run\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-sys-fs\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.208541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-dbus\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.208541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-kubernetes\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-var-lib-kubelet\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208499    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-lib-modules\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-modprobe-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-tuned\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-tmp\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-socket-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208611    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-conf\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208625    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2492dc3b-9893-45f5-af15-314312a0a18f-hosts-file\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2492dc3b-9893-45f5-af15-314312a0a18f-tmp-dir\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-kubelet-config\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-sys\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7zft\" (UniqueName: \"kubernetes.io/projected/7e65d887-3e39-400b-8a4f-7953a5a680e9-kube-api-access-c7zft\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysconfig\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208754    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-systemd\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-host\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-registration-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-device-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.208954 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bp5\" (UniqueName: \"kubernetes.io/projected/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kube-api-access-59bp5\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.208864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.208961    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.209008    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret podName:c6b17092-1d1d-40e0-b85f-28f05ab9f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:53.70899507 +0000 UTC m=+3.140180994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret") pod "global-pull-secret-syncer-lbvj4" (UID: "c6b17092-1d1d-40e0-b85f-28f05ab9f5f6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.209451    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-run\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.209511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-sys-fs\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.209552    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-dbus\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.209582    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-kubernetes\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.209630    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-var-lib-kubelet\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.209789 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.209746    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.210226 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.209824    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-lib-modules\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.210226 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.209876    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-modprobe-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.210330 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysconfig\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.210330 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210319    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2492dc3b-9893-45f5-af15-314312a0a18f-hosts-file\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.210773 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210683    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-sys\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.210773 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210716    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-kubelet-config\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.210773 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210724    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2492dc3b-9893-45f5-af15-314312a0a18f-tmp-dir\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.210999 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210788    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.210999 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210814    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-socket-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.210999 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210840    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-systemd\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.210999 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210879    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.210999 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210891    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-host\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.210999 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.210940    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-registration-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.211249 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.211010    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-device-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.211249 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.211072    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-conf\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.212545 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.212522    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-tuned\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.212756 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.212739    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-tmp\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.218514 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.218484    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mqhx\" (UniqueName: \"kubernetes.io/projected/2492dc3b-9893-45f5-af15-314312a0a18f-kube-api-access-8mqhx\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.219153 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.219128    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bp5\" (UniqueName: \"kubernetes.io/projected/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kube-api-access-59bp5\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.219245 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.219177    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7zft\" (UniqueName: \"kubernetes.io/projected/7e65d887-3e39-400b-8a4f-7953a5a680e9-kube-api-access-c7zft\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.294623 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.294586    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:00:53.303112 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.303091    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5sxfl"
Oct 30 21:00:53.307115 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:53.307088    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71f208a_7acb_47f5_b6c5_eb53347e18f9.slice/crio-3a4449d529242e23ff53efdd4d7c3db5a0ae65761c4fc988d378586c9ff94a9f WatchSource:0}: Error finding container 3a4449d529242e23ff53efdd4d7c3db5a0ae65761c4fc988d378586c9ff94a9f: Status 404 returned error can't find the container with id 3a4449d529242e23ff53efdd4d7c3db5a0ae65761c4fc988d378586c9ff94a9f
Oct 30 21:00:53.311401 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.311374    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:00:53.322054 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:53.322022    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595d39f4_1364_4e9e_87ae_09120d783bdd.slice/crio-5779f1266cd56af42347e94914cde8cdfe76b704fd47ef2747dc839222c81a7a WatchSource:0}: Error finding container 5779f1266cd56af42347e94914cde8cdfe76b704fd47ef2747dc839222c81a7a: Status 404 returned error can't find the container with id 5779f1266cd56af42347e94914cde8cdfe76b704fd47ef2747dc839222c81a7a
Oct 30 21:00:53.345193 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.345157    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:00:53.347311 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.347285    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:00:53.355841 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:53.355818    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93767113_7f48_4ed1_bb24_d6caaa025dfb.slice/crio-61429f4298006dd2eeab41049d2617ac951c50b0a9050cce97dd53062ae593b3 WatchSource:0}: Error finding container 61429f4298006dd2eeab41049d2617ac951c50b0a9050cce97dd53062ae593b3: Status 404 returned error can't find the container with id 61429f4298006dd2eeab41049d2617ac951c50b0a9050cce97dd53062ae593b3
Oct 30 21:00:53.358155 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:53.358132    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f1f33b_3e12_4a49_9ea7_918c9bf78265.slice/crio-2e80823a44905002c59d0b8415f29998656323ce3fab4ce6066e4b9ec4bdb98b WatchSource:0}: Error finding container 2e80823a44905002c59d0b8415f29998656323ce3fab4ce6066e4b9ec4bdb98b: Status 404 returned error can't find the container with id 2e80823a44905002c59d0b8415f29998656323ce3fab4ce6066e4b9ec4bdb98b
Oct 30 21:00:53.359334 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.359304    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:00:53.365335 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.365318    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:00:53.366643 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.366624    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:00:53.368683 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:53.368634    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e65d887_3e39_400b_8a4f_7953a5a680e9.slice/crio-2acb42d36f98ce77f7fe8ed346263515fc275558adecdd0003489ab6e00b3896 WatchSource:0}: Error finding container 2acb42d36f98ce77f7fe8ed346263515fc275558adecdd0003489ab6e00b3896: Status 404 returned error can't find the container with id 2acb42d36f98ce77f7fe8ed346263515fc275558adecdd0003489ab6e00b3896
Oct 30 21:00:53.373342 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.373261    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:00:53.374725 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:53.374658    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98364d6_40e1_4d54_ba37_84c14da2e2f3.slice/crio-2c8edaf727a7205225d32400d8cf7494e31c7a005d5ebd0d4e02ead46a22d98c WatchSource:0}: Error finding container 2c8edaf727a7205225d32400d8cf7494e31c7a005d5ebd0d4e02ead46a22d98c: Status 404 returned error can't find the container with id 2c8edaf727a7205225d32400d8cf7494e31c7a005d5ebd0d4e02ead46a22d98c
Oct 30 21:00:53.375853 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.375836    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:53.381924 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:53.381885    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2492dc3b_9893_45f5_af15_314312a0a18f.slice/crio-fb453be1021ff65de923b54bf6ae9d73c7585fd7b9b8a14677ba1be0648b19fb WatchSource:0}: Error finding container fb453be1021ff65de923b54bf6ae9d73c7585fd7b9b8a14677ba1be0648b19fb: Status 404 returned error can't find the container with id fb453be1021ff65de923b54bf6ae9d73c7585fd7b9b8a14677ba1be0648b19fb
Oct 30 21:00:53.386516 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:00:53.386492    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0130da1_910e_4623_834b_40b494b2309d.slice/crio-24752899e8cdadc52d3c7cce6a5ee7774808e50a0bd21fb69c61a6e35ea94cb9 WatchSource:0}: Error finding container 24752899e8cdadc52d3c7cce6a5ee7774808e50a0bd21fb69c61a6e35ea94cb9: Status 404 returned error can't find the container with id 24752899e8cdadc52d3c7cce6a5ee7774808e50a0bd21fb69c61a6e35ea94cb9
Oct 30 21:00:53.440380 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.440352    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:53.611814 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.611720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:53.611959 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.611875    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:53.611959 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.611951    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:54.611936267 +0000 UTC m=+4.043122203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:53.712295 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.712263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:53.712458 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:53.712315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:53.712538 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.712496    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:53.712538 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.712516    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:53.712538 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.712529    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5c8mg for pod openshift-network-diagnostics/network-check-target-kwb4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:53.712662 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.712587    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg podName:a6c067ea-93f5-4019-b9e4-100a1c8cb53c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:54.712568525 +0000 UTC m=+4.143754470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c8mg" (UniqueName: "kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg") pod "network-check-target-kwb4j" (UID: "a6c067ea-93f5-4019-b9e4-100a1c8cb53c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:53.713000 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.712972    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:53.713127 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:53.713024    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret podName:c6b17092-1d1d-40e0-b85f-28f05ab9f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:54.713010292 +0000 UTC m=+4.144196229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret") pod "global-pull-secret-syncer-lbvj4" (UID: "c6b17092-1d1d-40e0-b85f-28f05ab9f5f6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:54.039510 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.039448    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:55:53 +0000 UTC" deadline="2028-04-30 13:04:37.236779842 +0000 UTC"
Oct 30 21:00:54.039510 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.039509    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13144h3m43.197275585s"
Oct 30 21:00:54.072356 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.071827    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:54.072356 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.071986    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:00:54.095343 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.095024    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"24752899e8cdadc52d3c7cce6a5ee7774808e50a0bd21fb69c61a6e35ea94cb9"}
Oct 30 21:00:54.115049 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.114937    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9hs5s" event={"ID":"7e65d887-3e39-400b-8a4f-7953a5a680e9","Type":"ContainerStarted","Data":"2acb42d36f98ce77f7fe8ed346263515fc275558adecdd0003489ab6e00b3896"}
Oct 30 21:00:54.126531 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.126499    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wrffd" event={"ID":"595d39f4-1364-4e9e-87ae-09120d783bdd","Type":"ContainerStarted","Data":"5779f1266cd56af42347e94914cde8cdfe76b704fd47ef2747dc839222c81a7a"}
Oct 30 21:00:54.133181 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.133145    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmp52" event={"ID":"f71f208a-7acb-47f5-b6c5-eb53347e18f9","Type":"ContainerStarted","Data":"3a4449d529242e23ff53efdd4d7c3db5a0ae65761c4fc988d378586c9ff94a9f"}
Oct 30 21:00:54.137367 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.137341    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r7gpg" event={"ID":"2492dc3b-9893-45f5-af15-314312a0a18f","Type":"ContainerStarted","Data":"fb453be1021ff65de923b54bf6ae9d73c7585fd7b9b8a14677ba1be0648b19fb"}
Oct 30 21:00:54.143217 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.143190    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv" event={"ID":"a98364d6-40e1-4d54-ba37-84c14da2e2f3","Type":"ContainerStarted","Data":"2c8edaf727a7205225d32400d8cf7494e31c7a005d5ebd0d4e02ead46a22d98c"}
Oct 30 21:00:54.150748 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.150722    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" event={"ID":"66f1f33b-3e12-4a49-9ea7-918c9bf78265","Type":"ContainerStarted","Data":"2e80823a44905002c59d0b8415f29998656323ce3fab4ce6066e4b9ec4bdb98b"}
Oct 30 21:00:54.168276 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.168244    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r2kvg" event={"ID":"93767113-7f48-4ed1-bb24-d6caaa025dfb","Type":"ContainerStarted","Data":"61429f4298006dd2eeab41049d2617ac951c50b0a9050cce97dd53062ae593b3"}
Oct 30 21:00:54.184867 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.184827    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5sxfl" event={"ID":"0a6e387b-47f4-4966-b197-e2e73150d514","Type":"ContainerStarted","Data":"e4f6375d371e3b287305e9f95ad96b074998c08ffcf7c02fc876c095d6c61236"}
Oct 30 21:00:54.620629 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.620554    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:54.620835 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.620821    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:54.620896 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.620886    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:56.620868248 +0000 UTC m=+6.052054180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:54.721955 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.721340    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:54.721955 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:54.721397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:54.721955 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.721578    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:54.721955 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.721600    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:54.721955 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.721614    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5c8mg for pod openshift-network-diagnostics/network-check-target-kwb4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:54.721955 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.721785    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:54.721955 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.721901    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg podName:a6c067ea-93f5-4019-b9e4-100a1c8cb53c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:56.721655182 +0000 UTC m=+6.152841121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c8mg" (UniqueName: "kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg") pod "network-check-target-kwb4j" (UID: "a6c067ea-93f5-4019-b9e4-100a1c8cb53c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:54.731934 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:54.728719    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret podName:c6b17092-1d1d-40e0-b85f-28f05ab9f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:56.721921274 +0000 UTC m=+6.153107216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret") pod "global-pull-secret-syncer-lbvj4" (UID: "c6b17092-1d1d-40e0-b85f-28f05ab9f5f6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:55.039989 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:55.039922    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:55:53 +0000 UTC" deadline="2028-05-20 00:18:34.751864909 +0000 UTC"
Oct 30 21:00:55.039989 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:55.039987    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13611h17m39.71188243s"
Oct 30 21:00:55.073934 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:55.073256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:55.073934 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:55.073383    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:00:55.073934 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:55.073799    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:55.073934 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:55.073889    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:00:55.202858 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:55.202772    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"070c8221ace6d9936bb76d676bde1ea5","Type":"ContainerStarted","Data":"0f57e52b348d2ef635c82ee7cea6b6ed2dc97b0ec24614e9d2ce64b6ed639983"}
Oct 30 21:00:55.209380 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:55.208542    2574 generic.go:356] "Generic (PLEG): container finished" podID="c57946f0cf74e97be779f3304bd4cc78" containerID="471d7b98c95708bcf37f00231fa0320296d3bb927a3272a7219d1003d79abd55" exitCode=0
Oct 30 21:00:55.209380 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:55.208601    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"c57946f0cf74e97be779f3304bd4cc78","Type":"ContainerDied","Data":"471d7b98c95708bcf37f00231fa0320296d3bb927a3272a7219d1003d79abd55"}
Oct 30 21:00:55.216664 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:55.216618    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal" podStartSLOduration=3.216602412 podStartE2EDuration="3.216602412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:55.215720959 +0000 UTC m=+4.646906907" watchObservedRunningTime="2026-10-30 21:00:55.216602412 +0000 UTC m=+4.647788358"
Oct 30 21:00:56.071771 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:56.071246    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:56.071771 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.071380    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:00:56.214992 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:56.214959    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"c57946f0cf74e97be779f3304bd4cc78","Type":"ContainerStarted","Data":"0a48dacfd4fd3057d187f044e09e3e7c594f10b4a3084701d5537704d2f0cacc"}
Oct 30 21:00:56.231621 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:56.231560    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal" podStartSLOduration=4.231541279 podStartE2EDuration="4.231541279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.230967778 +0000 UTC m=+5.662153725" watchObservedRunningTime="2026-10-30 21:00:56.231541279 +0000 UTC m=+5.662727226"
Oct 30 21:00:56.640918 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:56.640835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:56.641087 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.641031    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:56.641154 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.641096    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:00.641077942 +0000 UTC m=+10.072263870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:56.741813 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:56.741777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:56.742120 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:56.741829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:56.742120 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.741942    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:56.742120 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.741959    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:56.742120 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.741984    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:56.742120 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.741998    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5c8mg for pod openshift-network-diagnostics/network-check-target-kwb4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:56.742120 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.742018    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret podName:c6b17092-1d1d-40e0-b85f-28f05ab9f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:00.741998162 +0000 UTC m=+10.173184090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret") pod "global-pull-secret-syncer-lbvj4" (UID: "c6b17092-1d1d-40e0-b85f-28f05ab9f5f6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:56.742120 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:56.742046    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg podName:a6c067ea-93f5-4019-b9e4-100a1c8cb53c nodeName:}" failed. No retries permitted until 2026-10-30 21:01:00.742031093 +0000 UTC m=+10.173217033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c8mg" (UniqueName: "kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg") pod "network-check-target-kwb4j" (UID: "a6c067ea-93f5-4019-b9e4-100a1c8cb53c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:57.074566 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:57.073834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:57.074566 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:57.073965    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:00:57.074566 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:57.074352    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:57.074566 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:57.074455    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:00:57.226369 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:57.224662    2574 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="ac09516245d8a4b8fe9de4d7a84d7ebd93bca51947b8a4ff9ffdee058d56643a" exitCode=0
Oct 30 21:00:57.226369 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:57.226116    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" event={"ID":"66f1f33b-3e12-4a49-9ea7-918c9bf78265","Type":"ContainerDied","Data":"ac09516245d8a4b8fe9de4d7a84d7ebd93bca51947b8a4ff9ffdee058d56643a"}
Oct 30 21:00:58.071399 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:58.071360    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:00:58.071573 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:58.071515    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:00:58.231981 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:58.231926    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r2kvg" event={"ID":"93767113-7f48-4ed1-bb24-d6caaa025dfb","Type":"ContainerStarted","Data":"8b384628f0e09a36b9d322e3df795b91f091cb20648d96c202e9a1225c80da71"}
Oct 30 21:00:59.072512 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:59.071809    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:00:59.072512 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:59.071935    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:00:59.072512 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:59.072355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:00:59.072512 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:00:59.072459    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:00:59.237906 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:59.237757    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wrffd" event={"ID":"595d39f4-1364-4e9e-87ae-09120d783bdd","Type":"ContainerStarted","Data":"da57f07bcc8d8e4e3312df618f90c60ccad4acce9bdeac6d7c7af2e3aebfaba4"}
Oct 30 21:00:59.251867 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:00:59.250804    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r2kvg" podStartSLOduration=4.569456267 podStartE2EDuration="8.250790859s" totalImagesPullingTime="3.681334592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:58.246960657 +0000 UTC m=+7.678146607" watchObservedRunningTime="2026-10-30 21:00:59.250790859 +0000 UTC m=+8.681976879"
Oct 30 21:01:00.071173 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.071144    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:00.071351 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.071277    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:00.243418 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.243382    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv" event={"ID":"a98364d6-40e1-4d54-ba37-84c14da2e2f3","Type":"ContainerStarted","Data":"62c1f9985db246291b6ed9b170d3b623edf5de3492a0e6061eaced302059ab03"}
Oct 30 21:01:00.666427 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.666388    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:01:00.667724 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.667535    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:01:00.673282 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.673257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:00.673463 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.673431    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:01:00.673515 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.673487    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:08.67347083 +0000 UTC m=+18.104656770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:01:00.695310 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.695256    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wrffd" podStartSLOduration=4.671510859 podStartE2EDuration="9.695235783s" totalImagesPullingTime="5.023724924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:59.251593419 +0000 UTC m=+8.682779359" watchObservedRunningTime="2026-10-30 21:01:00.695235783 +0000 UTC m=+10.126421730"
Oct 30 21:01:00.774713 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.774272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:00.774713 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.774324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:00.774937 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.774885    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:01:00.774988 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.774946    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret podName:c6b17092-1d1d-40e0-b85f-28f05ab9f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:08.774926421 +0000 UTC m=+18.206112359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret") pod "global-pull-secret-syncer-lbvj4" (UID: "c6b17092-1d1d-40e0-b85f-28f05ab9f5f6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:01:00.779278 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.779255    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:01:00.779278 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.779279    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:01:00.779415 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.779299    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5c8mg for pod openshift-network-diagnostics/network-check-target-kwb4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:01:00.779415 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:00.779377    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg podName:a6c067ea-93f5-4019-b9e4-100a1c8cb53c nodeName:}" failed. No retries permitted until 2026-10-30 21:01:08.779354664 +0000 UTC m=+18.210540611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c8mg" (UniqueName: "kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg") pod "network-check-target-kwb4j" (UID: "a6c067ea-93f5-4019-b9e4-100a1c8cb53c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:01:00.825754 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:00.824999    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:01:01.072005 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:01.071929    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:01.072149 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:01.072056    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:01.072149 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:01.072129    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:01.072266 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:01.072240    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:01.073760 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:01.073015    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:01:00.82502078Z","UUID":"2d9f3bd3-208b-466a-bf36-bf878c71f3dc","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:01:01.080967 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:01.080931    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:01:01.080967 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:01.080974    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:01:01.248166 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:01.248126    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv" event={"ID":"a98364d6-40e1-4d54-ba37-84c14da2e2f3","Type":"ContainerStarted","Data":"e6c0c8d2177493e2809d1e849db4cc3a1b13ca3c90204babb55f8b284ab24aaa"}
Oct 30 21:01:01.251186 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:01.251156    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5sxfl" event={"ID":"0a6e387b-47f4-4966-b197-e2e73150d514","Type":"ContainerStarted","Data":"3e72109f2fa3a4c51fc3aceaca6acda8a8aec642982e0436ef6777a827cc401c"}
Oct 30 21:01:01.276789 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:01.276732    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5sxfl" podStartSLOduration=2.504292388 podStartE2EDuration="10.276716024s" totalImagesPullingTime="7.772423636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:01.276685228 +0000 UTC m=+10.707871168" watchObservedRunningTime="2026-10-30 21:01:01.276716024 +0000 UTC m=+10.707901970"
Oct 30 21:01:02.071867 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:02.071641    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:02.072034 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:02.071997    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:02.255168 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:02.254700    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r7gpg" event={"ID":"2492dc3b-9893-45f5-af15-314312a0a18f","Type":"ContainerStarted","Data":"1436f0e5b4c643eed6bb7bdd0acc74864019f75138a7e01b5990474a7a9ddc13"}
Oct 30 21:01:02.259330 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:02.259292    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv" event={"ID":"a98364d6-40e1-4d54-ba37-84c14da2e2f3","Type":"ContainerStarted","Data":"171710e17574292ef4a6fb6c6c2679e11df051bd51b5cca5e4bb351afd341292"}
Oct 30 21:01:02.261555 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:02.261530    2574 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="240a49a5729c1f0aba876c6f66fda59cd38e75411f4783ebf9d11cfec86c5297" exitCode=0
Oct 30 21:01:02.261728 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:02.261706    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" event={"ID":"66f1f33b-3e12-4a49-9ea7-918c9bf78265","Type":"ContainerDied","Data":"240a49a5729c1f0aba876c6f66fda59cd38e75411f4783ebf9d11cfec86c5297"}
Oct 30 21:01:02.272343 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:02.272295    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r7gpg" podStartSLOduration=3.117722793 podStartE2EDuration="11.272282021s" totalImagesPullingTime="8.154559228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:02.271860752 +0000 UTC m=+11.703046700" watchObservedRunningTime="2026-10-30 21:01:02.272282021 +0000 UTC m=+11.703467966"
Oct 30 21:01:02.318256 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:02.318200    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv" podStartSLOduration=3.116932725 podStartE2EDuration="11.318184049s" totalImagesPullingTime="8.201251324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:02.317589154 +0000 UTC m=+11.748775157" watchObservedRunningTime="2026-10-30 21:01:02.318184049 +0000 UTC m=+11.749369994"
Oct 30 21:01:03.071864 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:03.071838    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:03.072022 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:03.071950    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:03.072240 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:03.071794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:03.072318 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:03.072303    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:03.266008 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:03.265920    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vmp52" event={"ID":"f71f208a-7acb-47f5-b6c5-eb53347e18f9","Type":"ContainerStarted","Data":"8b5512ce52185c01443d01359c449cc6963352a3dfbd8ce11f65c6fe16bf3d86"}
Oct 30 21:01:03.268523 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:03.268496    2574 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="d7a1fedabcc559c71826fc3a4aa60b9ff5780c6d383fd45db347348cf690cb26" exitCode=0
Oct 30 21:01:03.268655 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:03.268601    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" event={"ID":"66f1f33b-3e12-4a49-9ea7-918c9bf78265","Type":"ContainerDied","Data":"d7a1fedabcc559c71826fc3a4aa60b9ff5780c6d383fd45db347348cf690cb26"}
Oct 30 21:01:03.282656 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:03.282615    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vmp52" podStartSLOduration=4.453362921 podStartE2EDuration="12.282601752s" totalImagesPullingTime="7.829238831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:03.282466002 +0000 UTC m=+12.713651948" watchObservedRunningTime="2026-10-30 21:01:03.282601752 +0000 UTC m=+12.713787698"
Oct 30 21:01:04.071862 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:04.071829    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:04.072037 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:04.071960    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:04.279221 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:04.279182    2574 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="b534b69486be80baacc417c817ad44a599836e6e4389a72e5c7640f7d966149f" exitCode=0
Oct 30 21:01:04.279995 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:04.279266    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" event={"ID":"66f1f33b-3e12-4a49-9ea7-918c9bf78265","Type":"ContainerDied","Data":"b534b69486be80baacc417c817ad44a599836e6e4389a72e5c7640f7d966149f"}
Oct 30 21:01:04.281524 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:04.281494    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9hs5s" event={"ID":"7e65d887-3e39-400b-8a4f-7953a5a680e9","Type":"ContainerStarted","Data":"c10ba4bcf229d14b194c80bf0fa5d269e9ec136f6450152e23dbc7806889e3e9"}
Oct 30 21:01:04.326378 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:04.326290    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9hs5s" podStartSLOduration=3.376101418 podStartE2EDuration="13.326276095s" totalImagesPullingTime="9.950174677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:04.326061711 +0000 UTC m=+13.757247658" watchObservedRunningTime="2026-10-30 21:01:04.326276095 +0000 UTC m=+13.757462042"
Oct 30 21:01:05.074372 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:05.074334    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:05.074539 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:05.074491    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:05.075152 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:05.075133    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:05.075250 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:05.075208    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:06.072079 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:06.071524    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:06.072079 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:06.071708    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:07.071778 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:07.071501    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:07.071941 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:07.071578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:07.071941 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:07.071889    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:07.072024 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:07.071948    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:07.294171 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:07.294137    2574 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="4ffb9b36d18162a127d2d7df056562b547abd70c285adc3323b884d519462f29" exitCode=0
Oct 30 21:01:07.294645 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:07.294185    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" event={"ID":"66f1f33b-3e12-4a49-9ea7-918c9bf78265","Type":"ContainerDied","Data":"4ffb9b36d18162a127d2d7df056562b547abd70c285adc3323b884d519462f29"}
Oct 30 21:01:08.071628 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.071606    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:08.071743 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.071731    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:08.300708 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.300657    2574 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="56832e56cfb94ba29c640a86ec5b8dfb82eeacc9033b1eb3b145a3d206a96f15" exitCode=0
Oct 30 21:01:08.301200 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.300715    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" event={"ID":"66f1f33b-3e12-4a49-9ea7-918c9bf78265","Type":"ContainerDied","Data":"56832e56cfb94ba29c640a86ec5b8dfb82eeacc9033b1eb3b145a3d206a96f15"}
Oct 30 21:01:08.303910 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.303892    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:01:08.304236 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.304218    2574 generic.go:356] "Generic (PLEG): container finished" podID="e0130da1-910e-4623-834b-40b494b2309d" containerID="ce4ebfa129700630d7d77211f1270e584a58f9858d09ace1fbd8c917292329bf" exitCode=1
Oct 30 21:01:08.304315 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.304258    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"7178c330357672f1e6bb7214b7434e5e65a368060e08cea0cb5ccda0b7a3331c"}
Oct 30 21:01:08.304356 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.304320    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"6ab05b4b576c7809687fde162849ae1e7119376b186ebebd3b93d78a173a4d62"}
Oct 30 21:01:08.304443 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.304381    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"c03f654c2b6a0425c105e2172b0ee84ff3ad85713d1837ea836246f690bc2bbb"}
Oct 30 21:01:08.304443 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.304410    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"5526ac54c7e98293c1d28546d491a7b739cee2b0c3a8a395be8f8aa83624ebc2"}
Oct 30 21:01:08.304443 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.304438    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerDied","Data":"ce4ebfa129700630d7d77211f1270e584a58f9858d09ace1fbd8c917292329bf"}
Oct 30 21:01:08.304535 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.304465    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"0c4b3afca4e2c41643389b4ad70301c56cadd16fa4648a758290ff5fa8a92705"}
Oct 30 21:01:08.742044 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.741870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:08.742201 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.742007    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:01:08.742201 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.742138    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:24.742124685 +0000 UTC m=+34.173310610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:01:08.842807 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.842765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:08.842807 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:08.842808    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:08.843065 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.842910    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:01:08.843065 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.842917    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:01:08.843065 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.842935    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:01:08.843065 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.842945    2574 projected.go:196] Error preparing data for projected volume kube-api-access-5c8mg for pod openshift-network-diagnostics/network-check-target-kwb4j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:01:08.843065 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.842971    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret podName:c6b17092-1d1d-40e0-b85f-28f05ab9f5f6 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:24.842956595 +0000 UTC m=+34.274142519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret") pod "global-pull-secret-syncer-lbvj4" (UID: "c6b17092-1d1d-40e0-b85f-28f05ab9f5f6") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:01:08.843065 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:08.842986    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg podName:a6c067ea-93f5-4019-b9e4-100a1c8cb53c nodeName:}" failed. No retries permitted until 2026-10-30 21:01:24.842976162 +0000 UTC m=+34.274162089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5c8mg" (UniqueName: "kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg") pod "network-check-target-kwb4j" (UID: "a6c067ea-93f5-4019-b9e4-100a1c8cb53c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:01:09.072074 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:09.071999    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:09.072220 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:09.072108    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:09.072220 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:09.072166    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:09.072319 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:09.072274    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:09.309771 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:09.309738    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" event={"ID":"66f1f33b-3e12-4a49-9ea7-918c9bf78265","Type":"ContainerStarted","Data":"0d38a5798443aa3ff5e8eb301945f5e1a3f908ac5b8052c741782be334143963"}
Oct 30 21:01:09.333502 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:09.333407    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mfwjh" podStartSLOduration=7.919645367 podStartE2EDuration="18.333393964s" totalImagesPullingTime="10.413748597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:09.331779751 +0000 UTC m=+18.762965699" watchObservedRunningTime="2026-10-30 21:01:09.333393964 +0000 UTC m=+18.764579909"
Oct 30 21:01:10.071805 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:10.071777    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:10.071993 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:10.071899    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:11.072726 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:11.072699    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:11.073286 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:11.072801    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:11.073286 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:11.072849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:11.073286 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:11.072949    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:11.317239 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:11.317208    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:01:11.317567 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:11.317543    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"99b201c90ab2929945fd77f26f0c832cf9b20e6a201ca336cf76ee75d1057ab4"}
Oct 30 21:01:12.071149 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:12.071106    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:12.071315 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:12.071236    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:13.072110 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.072065    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:13.072841 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:13.072178    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:13.072841 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.072231    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:13.072841 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:13.072328    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:13.325213 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.325141    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:01:13.325517 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.325497    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"a3a39cb47a735068c15d1d2221c92ac7dcbeecf84a9a78544b6a170ea63634a3"}
Oct 30 21:01:13.325722 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.325706    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:13.325784 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.325772    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:13.325958 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.325939    2574 scope.go:122] "RemoveContainer" containerID="ce4ebfa129700630d7d77211f1270e584a58f9858d09ace1fbd8c917292329bf"
Oct 30 21:01:13.342348 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.342331    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:13.347187 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:13.347170    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:14.071313 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.071141    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:14.071487 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:14.071407    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:14.144241 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.144173    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gbg4d"]
Oct 30 21:01:14.148286 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.147753    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lbvj4"]
Oct 30 21:01:14.148286 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.147866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:14.148286 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:14.147974    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:14.148286 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.148275    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kwb4j"]
Oct 30 21:01:14.148560 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.148368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:14.148560 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:14.148465    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:14.331744 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.331715    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:01:14.332089 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.332069    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" event={"ID":"e0130da1-910e-4623-834b-40b494b2309d","Type":"ContainerStarted","Data":"183ecd2f1533566e82030cccf598437fb22f852c820d9bbcb8666856ddab78e3"}
Oct 30 21:01:14.332153 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.332138    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:14.332189 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.332179    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:14.332314 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:14.332296    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:14.376413 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:14.376361    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g" podStartSLOduration=8.873023501 podStartE2EDuration="23.376347346s" totalImagesPullingTime="14.503323845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:14.374507337 +0000 UTC m=+23.805693283" watchObservedRunningTime="2026-10-30 21:01:14.376347346 +0000 UTC m=+23.807533286"
Oct 30 21:01:15.334574 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:15.334548    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:15.891381 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:15.891340    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r2kvg_93767113-7f48-4ed1-bb24-d6caaa025dfb/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:16.054468 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:16.054432    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-r2kvg_93767113-7f48-4ed1-bb24-d6caaa025dfb/konnectivity-agent/0.log" limit=1024
Oct 30 21:01:16.071430 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:16.071398    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:16.071430 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:16.071429    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:16.071430 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:16.071431    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:16.071637 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:16.071496    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:16.071637 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:16.071552    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:16.071637 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:16.071615    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:16.464849 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:16.464807    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:18.071806 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:18.071764    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:18.071806 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:18.071792    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:18.071806 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:18.071801    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:18.072313 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:18.071889    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:18.072313 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:18.071939    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:18.072313 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:18.072034    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:19.659327 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:19.659294    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:01:19.659740 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:19.659452    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:19.659983 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:19.659959    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:01:20.071415 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.071381    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:20.071583 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:20.071474    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kwb4j" podUID="a6c067ea-93f5-4019-b9e4-100a1c8cb53c"
Oct 30 21:01:20.071583 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.071386    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:20.071583 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.071511    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:20.071718 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:20.071587    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lbvj4" podUID="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6"
Oct 30 21:01:20.071718 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:20.071694    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gbg4d" podUID="0ba46ac0-612c-4964-9454-8ae5e42bd728"
Oct 30 21:01:20.840043 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.839975    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeReady"
Oct 30 21:01:20.840397 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.840083    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:01:20.888245 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.888212    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jq4p6"]
Oct 30 21:01:20.891359 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.891340    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5l9cm"]
Oct 30 21:01:20.891473 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.891463    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:20.894300 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.894282    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:20.905104 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.905087    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:01:20.905199 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.905159    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:01:20.905247 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.905238    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s7wtw\""
Oct 30 21:01:20.905459 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.905444    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cp8br\""
Oct 30 21:01:20.906197 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.906179    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:01:20.906271 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.906185    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:01:20.906271 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.906209    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:01:20.909388 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.909361    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5l9cm"]
Oct 30 21:01:20.909488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:20.909426    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jq4p6"]
Oct 30 21:01:21.024025 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.023999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czh7s\" (UniqueName: \"kubernetes.io/projected/243d8065-2175-4a20-b229-ebaae8e987fb-kube-api-access-czh7s\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.024160 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.024031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/243d8065-2175-4a20-b229-ebaae8e987fb-config-volume\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.024160 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.024050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205f3154-c424-4dff-9536-cff842b7eac1-cert\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:21.024160 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.024076    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwlz\" (UniqueName: \"kubernetes.io/projected/205f3154-c424-4dff-9536-cff842b7eac1-kube-api-access-zxwlz\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:21.024269 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.024180    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243d8065-2175-4a20-b229-ebaae8e987fb-tmp-dir\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.024269 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.024218    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.124663 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.124599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243d8065-2175-4a20-b229-ebaae8e987fb-tmp-dir\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.124663 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.124652    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.124902 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.124704    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh7s\" (UniqueName: \"kubernetes.io/projected/243d8065-2175-4a20-b229-ebaae8e987fb-kube-api-access-czh7s\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.124902 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.124735    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/243d8065-2175-4a20-b229-ebaae8e987fb-config-volume\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.124902 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:21.124808    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:21.124902 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.124848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205f3154-c424-4dff-9536-cff842b7eac1-cert\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:21.124902 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:21.124877    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls podName:243d8065-2175-4a20-b229-ebaae8e987fb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:21.624854153 +0000 UTC m=+31.056040088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls") pod "dns-default-5l9cm" (UID: "243d8065-2175-4a20-b229-ebaae8e987fb") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:21.125147 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.124916    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwlz\" (UniqueName: \"kubernetes.io/projected/205f3154-c424-4dff-9536-cff842b7eac1-kube-api-access-zxwlz\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:21.125147 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.124975    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243d8065-2175-4a20-b229-ebaae8e987fb-tmp-dir\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.125241 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.125224    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/243d8065-2175-4a20-b229-ebaae8e987fb-config-volume\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.128587 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.128563    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205f3154-c424-4dff-9536-cff842b7eac1-cert\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:21.135827 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.135807    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh7s\" (UniqueName: \"kubernetes.io/projected/243d8065-2175-4a20-b229-ebaae8e987fb-kube-api-access-czh7s\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.135918 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.135861    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwlz\" (UniqueName: \"kubernetes.io/projected/205f3154-c424-4dff-9536-cff842b7eac1-kube-api-access-zxwlz\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:21.203334 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.203298    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:21.487134 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.487104    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jq4p6"]
Oct 30 21:01:21.490295 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:01:21.490260    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205f3154_c424_4dff_9536_cff842b7eac1.slice/crio-a3f28d5aa5d2cd3e609346aa08c4f1d0d96b3ed5c16287742813018be232b4d7 WatchSource:0}: Error finding container a3f28d5aa5d2cd3e609346aa08c4f1d0d96b3ed5c16287742813018be232b4d7: Status 404 returned error can't find the container with id a3f28d5aa5d2cd3e609346aa08c4f1d0d96b3ed5c16287742813018be232b4d7
Oct 30 21:01:21.627925 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:21.627889    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:21.628060 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:21.628003    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:21.628060 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:21.628054    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls podName:243d8065-2175-4a20-b229-ebaae8e987fb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:22.628040575 +0000 UTC m=+32.059226499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls") pod "dns-default-5l9cm" (UID: "243d8065-2175-4a20-b229-ebaae8e987fb") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:22.071120 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.071088    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:22.071572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.071098    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:22.071572 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.071105    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:22.075147 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.075120    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:01:22.075147 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.075134    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:01:22.075147 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.075124    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:01:22.075384 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.075319    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6m5dr\""
Oct 30 21:01:22.076449 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.076434    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:01:22.076555 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.076461    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6bj8t\""
Oct 30 21:01:22.076555 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.076493    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qd7pl\""
Oct 30 21:01:22.182359 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.182330    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rtfgg-7ntt6"]
Oct 30 21:01:22.195488 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.195462    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b7hnm"]
Oct 30 21:01:22.195648 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.195578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.200378 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.200331    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dcwvc\""
Oct 30 21:01:22.209809 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.209784    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rtfgg-7ntt6"]
Oct 30 21:01:22.209919 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.209828    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b7hnm"]
Oct 30 21:01:22.209919 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.209909    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.290506 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.290476    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:01:22.290506 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.290477    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:01:22.290506 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.290511    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:01:22.290791 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.290569    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:01:22.296137 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.296116    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:01:22.300728 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.300713    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:01:22.300813 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.300785    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpqzq\""
Oct 30 21:01:22.333087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.333087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333053    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/941b3d3d-7780-4b2d-8ac9-950122fe66a5-crio-socket\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.333087 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.333319 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpb8\" (UniqueName: \"kubernetes.io/projected/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-api-access-gfpb8\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.333319 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333105    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.333319 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333196    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzx6\" (UniqueName: \"kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.333443 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333344    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.333443 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333392    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.333499 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.333482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/941b3d3d-7780-4b2d-8ac9-950122fe66a5-data-volume\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.354806 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.354771    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jq4p6" event={"ID":"205f3154-c424-4dff-9536-cff842b7eac1","Type":"ContainerStarted","Data":"a3f28d5aa5d2cd3e609346aa08c4f1d0d96b3ed5c16287742813018be232b4d7"}
Oct 30 21:01:22.434730 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/941b3d3d-7780-4b2d-8ac9-950122fe66a5-data-volume\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.434730 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.434730 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/941b3d3d-7780-4b2d-8ac9-950122fe66a5-crio-socket\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.434730 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.434730 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpb8\" (UniqueName: \"kubernetes.io/projected/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-api-access-gfpb8\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzx6\" (UniqueName: \"kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.434916    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.435078    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/941b3d3d-7780-4b2d-8ac9-950122fe66a5-data-volume\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.435327    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.435380    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:22.935361326 +0000 UTC m=+32.366547253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.435383    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/941b3d3d-7780-4b2d-8ac9-950122fe66a5-crio-socket\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.435433    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.435475    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:22.935463545 +0000 UTC m=+32.366649495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : configmap "kube-rbac-proxy" not found
Oct 30 21:01:22.435543 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.435517    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.436133 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.435715    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.436133 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.435835    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.467244 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.467210    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzx6\" (UniqueName: \"kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.469265 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.469240    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpb8\" (UniqueName: \"kubernetes.io/projected/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-api-access-gfpb8\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.508088 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.508062    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"]
Oct 30 21:01:22.524799 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.524777    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:22.539753 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.539733    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"]
Oct 30 21:01:22.539867 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.539825    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"
Oct 30 21:01:22.547896 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.547868    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lw2cs\""
Oct 30 21:01:22.549619 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.549586    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:22.549775 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.549735    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:01:22.636655 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.636622    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:22.636794 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.636697    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v4cf\" (UniqueName: \"kubernetes.io/projected/3719ab51-43eb-4857-a169-3711b92d9206-kube-api-access-4v4cf\") pod \"migrator-66df5d496b-l56np\" (UID: \"3719ab51-43eb-4857-a169-3711b92d9206\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"
Oct 30 21:01:22.636847 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.636822    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:22.636918 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.636905    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls podName:243d8065-2175-4a20-b229-ebaae8e987fb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:24.636882579 +0000 UTC m=+34.068068519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls") pod "dns-default-5l9cm" (UID: "243d8065-2175-4a20-b229-ebaae8e987fb") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:22.663927 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.663893    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d2r2t"]
Oct 30 21:01:22.683231 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.683209    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d2r2t"]
Oct 30 21:01:22.683362 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.683303    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.687339 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.687312    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:01:22.687448 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.687397    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k54sx\""
Oct 30 21:01:22.687820 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.687799    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:01:22.687820 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.687809    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:01:22.687978 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.687857    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:01:22.737491 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.737454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4cf\" (UniqueName: \"kubernetes.io/projected/3719ab51-43eb-4857-a169-3711b92d9206-kube-api-access-4v4cf\") pod \"migrator-66df5d496b-l56np\" (UID: \"3719ab51-43eb-4857-a169-3711b92d9206\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"
Oct 30 21:01:22.747091 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.747063    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4cf\" (UniqueName: \"kubernetes.io/projected/3719ab51-43eb-4857-a169-3711b92d9206-kube-api-access-4v4cf\") pod \"migrator-66df5d496b-l56np\" (UID: \"3719ab51-43eb-4857-a169-3711b92d9206\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"
Oct 30 21:01:22.826329 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.826296    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rtfgg-7ntt6"]
Oct 30 21:01:22.831434 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:01:22.831404    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c66adeb_03be_476e_960e_2544b9605375.slice/crio-98cb2cc90ab939e3a55004b4d377ecabba05d2f7203fc8b4ca4f4d335f7826d3 WatchSource:0}: Error finding container 98cb2cc90ab939e3a55004b4d377ecabba05d2f7203fc8b4ca4f4d335f7826d3: Status 404 returned error can't find the container with id 98cb2cc90ab939e3a55004b4d377ecabba05d2f7203fc8b4ca4f4d335f7826d3
Oct 30 21:01:22.838690 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.838643    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/235a4809-212c-45d1-b522-d71ab502e721-signing-key\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.838783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.838718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/235a4809-212c-45d1-b522-d71ab502e721-signing-cabundle\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.838783 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.838736    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmct\" (UniqueName: \"kubernetes.io/projected/235a4809-212c-45d1-b522-d71ab502e721-kube-api-access-2zmct\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.878453 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.878381    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"
Oct 30 21:01:22.939382 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.939344    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.939531 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.939458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/235a4809-212c-45d1-b522-d71ab502e721-signing-key\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.939531 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.939505    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:22.939531 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.939514    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:22.939660 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.939536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/235a4809-212c-45d1-b522-d71ab502e721-signing-cabundle\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.939660 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.939561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmct\" (UniqueName: \"kubernetes.io/projected/235a4809-212c-45d1-b522-d71ab502e721-kube-api-access-2zmct\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.939660 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.939585    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.939570567 +0000 UTC m=+33.370756525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:22.939913 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.939895    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:01:22.939985 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:22.939950    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.939935082 +0000 UTC m=+33.371121020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : configmap "kube-rbac-proxy" not found
Oct 30 21:01:22.940639 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.940598    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/235a4809-212c-45d1-b522-d71ab502e721-signing-cabundle\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.942866 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.942844    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/235a4809-212c-45d1-b522-d71ab502e721-signing-key\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:22.949181 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:22.949162    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmct\" (UniqueName: \"kubernetes.io/projected/235a4809-212c-45d1-b522-d71ab502e721-kube-api-access-2zmct\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:23.006963 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:23.006932    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:23.183541 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:23.183513    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"]
Oct 30 21:01:23.187909 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:01:23.187879    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3719ab51_43eb_4857_a169_3711b92d9206.slice/crio-fd9449a3541a4af2bc895ccd6e7428e0143d3707af8b781f4a8a633c9275c2d8 WatchSource:0}: Error finding container fd9449a3541a4af2bc895ccd6e7428e0143d3707af8b781f4a8a633c9275c2d8: Status 404 returned error can't find the container with id fd9449a3541a4af2bc895ccd6e7428e0143d3707af8b781f4a8a633c9275c2d8
Oct 30 21:01:23.314985 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:01:23.314948    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235a4809_212c_45d1_b522_d71ab502e721.slice/crio-691138e98a4e6288cf8a058d70defffe27be7d9148776438227954ae5242b691 WatchSource:0}: Error finding container 691138e98a4e6288cf8a058d70defffe27be7d9148776438227954ae5242b691: Status 404 returned error can't find the container with id 691138e98a4e6288cf8a058d70defffe27be7d9148776438227954ae5242b691
Oct 30 21:01:23.317037 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:23.317014    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-d2r2t"]
Oct 30 21:01:23.358019 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:23.357991    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t" event={"ID":"235a4809-212c-45d1-b522-d71ab502e721","Type":"ContainerStarted","Data":"691138e98a4e6288cf8a058d70defffe27be7d9148776438227954ae5242b691"}
Oct 30 21:01:23.359053 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:23.359027    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6" event={"ID":"1c66adeb-03be-476e-960e-2544b9605375","Type":"ContainerStarted","Data":"98cb2cc90ab939e3a55004b4d377ecabba05d2f7203fc8b4ca4f4d335f7826d3"}
Oct 30 21:01:23.360246 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:23.360213    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np" event={"ID":"3719ab51-43eb-4857-a169-3711b92d9206","Type":"ContainerStarted","Data":"fd9449a3541a4af2bc895ccd6e7428e0143d3707af8b781f4a8a633c9275c2d8"}
Oct 30 21:01:23.948143 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:23.948102    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:23.948309 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:23.948214    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:23.948385 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:23.948324    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:01:23.948385 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:23.948384    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:25.948365847 +0000 UTC m=+35.379551784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : configmap "kube-rbac-proxy" not found
Oct 30 21:01:23.948848 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:23.948825    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:23.948942 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:23.948881    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:25.948865883 +0000 UTC m=+35.380051820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:24.654530 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:24.654488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:24.655071 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:24.654620    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:24.655071 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:24.654709    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls podName:243d8065-2175-4a20-b229-ebaae8e987fb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.654689653 +0000 UTC m=+38.085875577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls") pod "dns-default-5l9cm" (UID: "243d8065-2175-4a20-b229-ebaae8e987fb") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:24.755255 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:24.755172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:24.755389 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:24.755360    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:01:24.755442 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:24.755419    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:56.755404615 +0000 UTC m=+66.186590540 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : secret "metrics-daemon-secret" not found
Oct 30 21:01:24.855945 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:24.855915    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:24.856059 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:24.855967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:24.859070 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:24.859045    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:24.860217 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:24.860170    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:25.101344 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.101258    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:25.112033 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.111997    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:25.368767 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.368650    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np" event={"ID":"3719ab51-43eb-4857-a169-3711b92d9206","Type":"ContainerStarted","Data":"44aa68ca2ae8bac3dd5730593deabec1f199f8ef28a0e54ad78e37213cc7f849"}
Oct 30 21:01:25.370417 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.370391    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jq4p6" event={"ID":"205f3154-c424-4dff-9536-cff842b7eac1","Type":"ContainerStarted","Data":"df8326677ecfeee7cc2ffe5eb0894873f3520da029fea7adc2fd30359356c9d9"}
Oct 30 21:01:25.384605 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.384555    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np" podStartSLOduration=1.4131213 podStartE2EDuration="3.384540976s" totalImagesPullingTime="1.971419676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:25.38416038 +0000 UTC m=+34.815346330" watchObservedRunningTime="2026-10-30 21:01:25.384540976 +0000 UTC m=+34.815726923"
Oct 30 21:01:25.400525 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.400475    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jq4p6" podStartSLOduration=2.050102525 podStartE2EDuration="5.400457984s" totalImagesPullingTime="3.350355459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:25.400233499 +0000 UTC m=+34.831419453" watchObservedRunningTime="2026-10-30 21:01:25.400457984 +0000 UTC m=+34.831643930"
Oct 30 21:01:25.439468 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.439439    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lbvj4"]
Oct 30 21:01:25.441995 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.441971    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kwb4j"]
Oct 30 21:01:25.442433 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:01:25.442384    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b17092_1d1d_40e0_b85f_28f05ab9f5f6.slice/crio-d4190d7ce70853a4046e9f978d93b93f5498f879af3183b4d3472127e3cdcb08 WatchSource:0}: Error finding container d4190d7ce70853a4046e9f978d93b93f5498f879af3183b4d3472127e3cdcb08: Status 404 returned error can't find the container with id d4190d7ce70853a4046e9f978d93b93f5498f879af3183b4d3472127e3cdcb08
Oct 30 21:01:25.445090 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:01:25.445066    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c067ea_93f5_4019_b9e4_100a1c8cb53c.slice/crio-4c7c0bebb2659e387b6cb9ab840be9da3979aa3c86d7597aff85df8d7c963cc2 WatchSource:0}: Error finding container 4c7c0bebb2659e387b6cb9ab840be9da3979aa3c86d7597aff85df8d7c963cc2: Status 404 returned error can't find the container with id 4c7c0bebb2659e387b6cb9ab840be9da3979aa3c86d7597aff85df8d7c963cc2
Oct 30 21:01:25.963914 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.963885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:25.964294 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:25.964012    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:01:25.964294 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:25.964013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:25.964294 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:25.964101    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.964076387 +0000 UTC m=+39.395262331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : configmap "kube-rbac-proxy" not found
Oct 30 21:01:25.964294 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:25.964252    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:25.964294 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:25.964298    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.964285457 +0000 UTC m=+39.395471388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:26.375603 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:26.375338    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6" event={"ID":"1c66adeb-03be-476e-960e-2544b9605375","Type":"ContainerStarted","Data":"ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079"}
Oct 30 21:01:26.377209 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:26.377175    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lbvj4" event={"ID":"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6","Type":"ContainerStarted","Data":"d4190d7ce70853a4046e9f978d93b93f5498f879af3183b4d3472127e3cdcb08"}
Oct 30 21:01:26.378847 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:26.378823    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kwb4j" event={"ID":"a6c067ea-93f5-4019-b9e4-100a1c8cb53c","Type":"ContainerStarted","Data":"4c7c0bebb2659e387b6cb9ab840be9da3979aa3c86d7597aff85df8d7c963cc2"}
Oct 30 21:01:26.396868 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:26.396822    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6" podStartSLOduration=1.460401122 podStartE2EDuration="4.396805724s" totalImagesPullingTime="2.936404602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:26.396010164 +0000 UTC m=+35.827196111" watchObservedRunningTime="2026-10-30 21:01:26.396805724 +0000 UTC m=+35.827991674"
Oct 30 21:01:27.383812 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:27.383743    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t" event={"ID":"235a4809-212c-45d1-b522-d71ab502e721","Type":"ContainerStarted","Data":"a9b912a2a3b0b901ff45f51804e984c98eb06d7899d243b4187e29488e503543"}
Oct 30 21:01:27.403797 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:27.403738    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t" podStartSLOduration=2.412075037 podStartE2EDuration="5.403713064s" totalImagesPullingTime="2.991638027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:27.401442395 +0000 UTC m=+36.832628340" watchObservedRunningTime="2026-10-30 21:01:27.403713064 +0000 UTC m=+36.834899012"
Oct 30 21:01:28.600967 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:28.600938    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r7gpg_2492dc3b-9893-45f5-af15-314312a0a18f/dns-node-resolver/0.log"
Oct 30 21:01:28.686504 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:28.686465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:28.689096 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:28.689071    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:28.711078 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:28.711048    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:29.001405 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:29.001374    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5l9cm"]
Oct 30 21:01:29.005434 ip-10-0-140-12 kubenswrapper[2574]: W1030 21:01:29.005397    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243d8065_2175_4a20_b229_ebaae8e987fb.slice/crio-61860744563f71403a2c860becf15b89d7012cb4ad54301720b902d710a61aa7 WatchSource:0}: Error finding container 61860744563f71403a2c860becf15b89d7012cb4ad54301720b902d710a61aa7: Status 404 returned error can't find the container with id 61860744563f71403a2c860becf15b89d7012cb4ad54301720b902d710a61aa7
Oct 30 21:01:29.390094 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:29.390007    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5l9cm" event={"ID":"243d8065-2175-4a20-b229-ebaae8e987fb","Type":"ContainerStarted","Data":"61860744563f71403a2c860becf15b89d7012cb4ad54301720b902d710a61aa7"}
Oct 30 21:01:29.601033 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:29.600998    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wrffd_595d39f4-1364-4e9e-87ae-09120d783bdd/node-ca/0.log"
Oct 30 21:01:30.001152 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:30.001127    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jq4p6_205f3154-c424-4dff-9536-cff842b7eac1/serve-healthcheck-canary/0.log"
Oct 30 21:01:30.006654 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:30.006628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:30.006814 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:30.006710    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:30.007281 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:30.007253    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:30.007394 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:30.007365    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:30.007452 ip-10-0-140-12 kubenswrapper[2574]: E1030 21:01:30.007425    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:38.007406939 +0000 UTC m=+47.438592877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:30.395751 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:30.395235    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kwb4j" event={"ID":"a6c067ea-93f5-4019-b9e4-100a1c8cb53c","Type":"ContainerStarted","Data":"0f037a8c500e0e3f0575318ef1d6d52b2a79cc6e839b24fa2fbca618ff63a364"}
Oct 30 21:01:30.395751 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:30.395524    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:30.602663 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:30.602632    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-l56np_3719ab51-43eb-4857-a169-3711b92d9206/migrator/0.log"
Oct 30 21:01:31.401146 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:31.401109    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lbvj4" event={"ID":"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6","Type":"ContainerStarted","Data":"c5343901bd53964aee63dfd1f6854a60569d1c85ce161e0a050251fbea6ee86a"}
Oct 30 21:01:31.404057 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:31.404024    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5l9cm" event={"ID":"243d8065-2175-4a20-b229-ebaae8e987fb","Type":"ContainerStarted","Data":"8acdce1c111591e45a225ad3d5b022e102f34b0a2422de5a2d07ba459ebbc0ef"}
Oct 30 21:01:31.420870 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:31.420558    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lbvj4" podStartSLOduration=34.668696048 podStartE2EDuration="40.420544s" totalImagesPullingTime="5.751847952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:31.420019431 +0000 UTC m=+40.851205376" watchObservedRunningTime="2026-10-30 21:01:31.420544 +0000 UTC m=+40.851729937"
Oct 30 21:01:31.421408 ip-10-0-140-12 kubenswrapper[2574]: I1030 21:01:31.421377    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kwb4j" podStartSLOduration=36.241766902 podStartE2EDuration="40.421366306s" totalImagesPullingTime="4.179599404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:30.411133628 +0000 UTC m=+39.842319568" watchObservedRunningTime="2026-10-30 21:01:31.421366306 +0000 UTC m=+40.852552251"
Oct 30 21:01:31.590143 ip-10-0-140-12 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:01:31.607159 ip-10-0-140-12 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:01:31.607339 ip-10-0-140-12 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:01:31.607395 ip-10-0-140-12 systemd[1]: kubelet.service: Consumed 2.518s CPU time.
Oct 30 21:01:31.618386 ip-10-0-140-12 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:01:31.705148 ip-10-0-140-12 kubenswrapper[4633]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:01:31.705148 ip-10-0-140-12 kubenswrapper[4633]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:01:31.705148 ip-10-0-140-12 kubenswrapper[4633]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:01:31.705148 ip-10-0-140-12 kubenswrapper[4633]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:01:31.707604 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707481    4633 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:31.707604 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707604    4633 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707610    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707616    4633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707621    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707625    4633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707630    4633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707634    4633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707639    4633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707643    4633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707647    4633 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707649    4633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707659    4633 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707678    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707683    4633 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707688    4633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707693    4633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707697    4633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707701    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707708    4633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:31.707744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707715    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707720    4633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707733    4633 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707737    4633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707741    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707746    4633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707751    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707755    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707780    4633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707786    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707790    4633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707795    4633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707799    4633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707803    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707808    4633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707812    4633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707817    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707821    4633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707832    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:31.708286 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707836    4633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707839    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707842    4633 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707844    4633 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707847    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707849    4633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707852    4633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707856    4633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707860    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707866    4633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707871    4633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707877    4633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707881    4633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707886    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707890    4633 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707900    4633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707905    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707908    4633 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707911    4633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707913    4633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:31.708769 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707916    4633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707919    4633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707922    4633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707924    4633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707927    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707930    4633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707932    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707935    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707938    4633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707940    4633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707943    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707946    4633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707948    4633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707951    4633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707954    4633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707957    4633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707959    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707963    4633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707965    4633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:31.709248 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707968    4633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707971    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707991    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.707996    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708000    4633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708002    4633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708005    4633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708008    4633 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708011    4633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708014    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708017    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708020    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708022    4633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708025    4633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708028    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708031    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708034    4633 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708038    4633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708040    4633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708043    4633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:31.709765 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708046    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708049    4633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708052    4633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708055    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708057    4633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708062    4633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708066    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708071    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708961    4633 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708969    4633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708972    4633 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708975    4633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708978    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708981    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708984    4633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708988    4633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708991    4633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708994    4633 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.708997    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:31.710251 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709000    4633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709003    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709005    4633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709010    4633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709014    4633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709017    4633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709021    4633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709024    4633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709026    4633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709029    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709032    4633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709034    4633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709037    4633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709040    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709043    4633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709047    4633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709052    4633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709055    4633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709058    4633 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:31.710754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709061    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709064    4633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709067    4633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709070    4633 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709073    4633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709075    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709078    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709082    4633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709085    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709087    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709089    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709093    4633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709095    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709098    4633 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709100    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709103    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709106    4633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709109    4633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709111    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:31.711220 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709114    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709116    4633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709119    4633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709122    4633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709124    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709127    4633 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709129    4633 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709132    4633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709135    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709137    4633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709140    4633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709144    4633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709150    4633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709153    4633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709156    4633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709159    4633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709161    4633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709164    4633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709167    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709170    4633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:31.711684 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709172    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709175    4633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709178    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709180    4633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709183    4633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709186    4633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709188    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709191    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709194    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709197    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709199    4633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709202    4633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709205    4633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709207    4633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709210    4633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709213    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709216    4633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709219    4633 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709221    4633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:31.712219 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709224    4633 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709227    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709230    4633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709232    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709235    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709240    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709243    4633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709245    4633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709248    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709272    4633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709275    4633 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709286    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709289    4633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709292    4633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709294    4633 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709297    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709299    4633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709302    4633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709859    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709865    4633 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:31.712693 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709868    4633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709871    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709874    4633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709879    4633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709884    4633 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709887    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709890    4633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709893    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709896    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709899    4633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709903    4633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709907    4633 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709910    4633 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709914    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709917    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709919    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709922    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709925    4633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709928    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:31.713177 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709931    4633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709934    4633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709936    4633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709939    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709942    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709944    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709947    4633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709950    4633 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709952    4633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709955    4633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709958    4633 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709960    4633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709963    4633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709966    4633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709969    4633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709972    4633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709975    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709978    4633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709980    4633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709983    4633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:31.713838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709985    4633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709988    4633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709991    4633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709994    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709996    4633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.709999    4633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710001    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710004    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710006    4633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710009    4633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710012    4633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710014    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710017    4633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710020    4633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710022    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710025    4633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710027    4633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710030    4633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710033    4633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:31.714354 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710035    4633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710037    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710040    4633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710043    4633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710045    4633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710048    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710057    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710060    4633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710063    4633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710066    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710069    4633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710071    4633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710074    4633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710076    4633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710079    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710082    4633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710084    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710087    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710089    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710092    4633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:31.714884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710095    4633 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710097    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710100    4633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710103    4633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710105    4633 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710108    4633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710111    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710113    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710116    4633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710118    4633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710121    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710123    4633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710126    4633 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710128    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710131    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710133    4633 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710136    4633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710138    4633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710141    4633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710151    4633 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:31.715372 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710154    4633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710156    4633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710159    4633 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710161    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710164    4633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710166    4633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710285    4633 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710304    4633 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710314    4633 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710323    4633 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710328    4633 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710331    4633 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710336    4633 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710340    4633 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710344    4633 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710347    4633 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710351    4633 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710354    4633 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710357    4633 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710361    4633 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710364    4633 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710367    4633 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710370    4633 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710373    4633 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:01:31.715889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710377    4633 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710380    4633 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710383    4633 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710386    4633 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710390    4633 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710394    4633 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710397    4633 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710400    4633 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710405    4633 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710408    4633 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710411    4633 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710414    4633 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710418    4633 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710421    4633 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710427    4633 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710430    4633 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710433    4633 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710438    4633 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710441    4633 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710444    4633 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710450    4633 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710453    4633 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710456    4633 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710459    4633 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:01:31.716525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710462    4633 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710468    4633 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710471    4633 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710474    4633 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710478    4633 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710481    4633 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710484    4633 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710487    4633 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710490    4633 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710493    4633 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710496    4633 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710499    4633 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710503    4633 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710506    4633 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710510    4633 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710513    4633 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710516    4633 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710520    4633 flags.go:64] FLAG: --help="false"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710523    4633 flags.go:64] FLAG: --hostname-override="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710526    4633 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710529    4633 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710532    4633 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710537    4633 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:01:31.717130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710541    4633 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710544    4633 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710550    4633 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710553    4633 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710556    4633 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710559    4633 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710563    4633 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710566    4633 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710569    4633 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710572    4633 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710576    4633 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710579    4633 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710582    4633 flags.go:64] FLAG: --lock-file=""
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710585    4633 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710588    4633 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710591    4633 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710596    4633 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710600    4633 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710603    4633 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710605    4633 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710609    4633 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710612    4633 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710615    4633 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710618    4633 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710623    4633 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:01:31.717719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710626    4633 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710630    4633 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710634    4633 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710637    4633 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710640    4633 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710643    4633 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710648    4633 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710651    4633 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710654    4633 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710663    4633 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710678    4633 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710682    4633 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710685    4633 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710689    4633 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710692    4633 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710695    4633 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710698    4633 flags.go:64] FLAG: --port="10250"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710701    4633 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710704    4633 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0859b3f8478b12bef"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710708    4633 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710711    4633 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710714    4633 flags.go:64] FLAG: --register-node="true"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710717    4633 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710721    4633 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710724    4633 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710727    4633 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:01:31.718313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710730    4633 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710734    4633 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710737    4633 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710740    4633 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710743    4633 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710746    4633 flags.go:64] FLAG: --runonce="false"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710749    4633 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710752    4633 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710756    4633 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710759    4633 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710762    4633 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710766    4633 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710770    4633 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710773    4633 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710776    4633 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710780    4633 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710783    4633 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710786    4633 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710790    4633 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710793    4633 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710796    4633 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710802    4633 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710805    4633 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710808    4633 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710812    4633 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:01:31.718984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710815    4633 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710817    4633 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710821    4633 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710824    4633 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710827    4633 flags.go:64] FLAG: --v="2"
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710831    4633 flags.go:64] FLAG: --version="false"
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710835    4633 flags.go:64] FLAG: --vmodule=""
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710839    4633 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.710843    4633 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.710999    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711003    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711006    4633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711009    4633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711011    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711014    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711017    4633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711019    4633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711022    4633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711025    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711027    4633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711030    4633 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711032    4633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:31.719576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711036    4633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711039    4633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711041    4633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711044    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711046    4633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711049    4633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711051    4633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711054    4633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711057    4633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711059    4633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711062    4633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711064    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711067    4633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711069    4633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711072    4633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711074    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711077    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711080    4633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711083    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:31.720143 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711085    4633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711088    4633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711090    4633 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711093    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711096    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711098    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711101    4633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711103    4633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711106    4633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711109    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711111    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711114    4633 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711117    4633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711121    4633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711124    4633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711126    4633 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711129    4633 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711131    4633 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711134    4633 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711136    4633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:31.720654 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711139    4633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711141    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711144    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711147    4633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711149    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711152    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711154    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711157    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711160    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711162    4633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711165    4633 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711168    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711170    4633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711173    4633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711175    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711178    4633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711180    4633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711182    4633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711185    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:31.721430 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711188    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711190    4633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711193    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711196    4633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711199    4633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711201    4633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711206    4633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711208    4633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711212    4633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711216    4633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711220    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711223    4633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711226    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711229    4633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711232    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711234    4633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711239    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711242    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711245    4633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:31.722066 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711247    4633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711250    4633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711252    4633 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711255    4633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711258    4633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711260    4633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711263    4633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711265    4633 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711268    4633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711271    4633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711273    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711276    4633 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711278    4633 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711281    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711284    4633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711286    4633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:31.722621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.711312    4633 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711359    4633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711364    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711369    4633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711372    4633 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711374    4633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711377    4633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711380    4633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711382    4633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711385    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711387    4633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711390    4633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711393    4633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711396    4633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711401    4633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711403    4633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711406    4633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711408    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711411    4633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:31.723230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711414    4633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711416    4633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711419    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711422    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711424    4633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711427    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711430    4633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711432    4633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711435    4633 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711437    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711440    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711443    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711446    4633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711448    4633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711451    4633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711454    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711461    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711464    4633 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711467    4633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711469    4633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:31.723919 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711472    4633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711475    4633 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711477    4633 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711480    4633 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711482    4633 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711485    4633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711487    4633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711491    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711494    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711497    4633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711499    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711502    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711504    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711507    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711510    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711513    4633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711516    4633 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711518    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711521    4633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:31.724428 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711524    4633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711527    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711529    4633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711532    4633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711534    4633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711537    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711539    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711542    4633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711544    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711549    4633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711552    4633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711555    4633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711558    4633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711561    4633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711564    4633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711569    4633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711572    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711575    4633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711578    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:31.725093 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711580    4633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711584    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711587    4633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711590    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711593    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711595    4633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711598    4633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711601    4633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711603    4633 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711606    4633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711609    4633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711611    4633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711614    4633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711616    4633 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711619    4633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711621    4633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711624    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711626    4633 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711629    4633 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711631    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:31.725642 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711634    4633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711637    4633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711640    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711644    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711647    4633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711650    4633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711653    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711655    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711658    4633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.711660    4633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.715570    4633 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.715587    4633 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.715606    4633 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.715612    4633 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715703    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715710    4633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715713    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715717    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715720    4633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715723    4633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:31.726215 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715732    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715735    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715738    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715741    4633 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715744    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715747    4633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715749    4633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715753    4633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715755    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715758    4633 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715761    4633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715763    4633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715766    4633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715769    4633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715771    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715774    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715777    4633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715779    4633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715782    4633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:31.726754 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715784    4633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715787    4633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715789    4633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715792    4633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715796    4633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715799    4633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715802    4633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715805    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715808    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715810    4633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715813    4633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715815    4633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715818    4633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715821    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715824    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715827    4633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715832    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715835    4633 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715839    4633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:31.727230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715842    4633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715845    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715848    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715851    4633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715854    4633 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715856    4633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715859    4633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715862    4633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715864    4633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715867    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715870    4633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715873    4633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715876    4633 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715878    4633 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715881    4633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715883    4633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715886    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715889    4633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715892    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:31.727845 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715897    4633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715901    4633 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715905    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715908    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715910    4633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715913    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715916    4633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715918    4633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715921    4633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715923    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715925    4633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715928    4633 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715931    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715934    4633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715936    4633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715939    4633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715941    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715944    4633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715946    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715949    4633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:31.728302 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715952    4633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715955    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715957    4633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715960    4633 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715962    4633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715965    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715967    4633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715970    4633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715973    4633 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715976    4633 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715978    4633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715981    4633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715984    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715987    4633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715990    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715992    4633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715995    4633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.715997    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716000    4633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716002    4633 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:31.728854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716005    4633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716007    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716010    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.716037    4633 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716185    4633 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716190    4633 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716193    4633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716196    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716198    4633 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716201    4633 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716204    4633 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716206    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716209    4633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716212    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716216    4633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716220    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:31.729380 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716224    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716228    4633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716231    4633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716233    4633 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716236    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716239    4633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716241    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716244    4633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716247    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716250    4633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716252    4633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716255    4633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716258    4633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716260    4633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716264    4633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716267    4633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716270    4633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716274    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716276    4633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:31.729819 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716279    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716282    4633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716284    4633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716286    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716289    4633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716292    4633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716294    4633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716297    4633 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716299    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716302    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716304    4633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716307    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716309    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716312    4633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716314    4633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716317    4633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716320    4633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716322    4633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716325    4633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716327    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:31.730534 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716330    4633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716332    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716335    4633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716338    4633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716341    4633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716343    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716346    4633 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716348    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716351    4633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716353    4633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716356    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716358    4633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716361    4633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716363    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716366    4633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716369    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716371    4633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716373    4633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716376    4633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716378    4633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:31.731227 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716381    4633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716383    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716387    4633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716391    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716395    4633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716398    4633 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716401    4633 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716404    4633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716406    4633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716409    4633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716412    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716414    4633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716417    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716419    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716421    4633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716424    4633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716427    4633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716429    4633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716432    4633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:31.732199 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716434    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716437    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716439    4633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716442    4633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716444    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716447    4633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716449    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716452    4633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716454    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716456    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716459    4633 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716461    4633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716464    4633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716466    4633 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716469    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716472    4633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:31.732914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.716497    4633 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716527    4633 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716531    4633 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716533    4633 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716536    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716539    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716542    4633 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716544    4633 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716547    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716549    4633 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716552    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716554    4633 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716557    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716559    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716563    4633 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716565    4633 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716568    4633 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716570    4633 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716573    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:31.733321 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716575    4633 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716578    4633 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716580    4633 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716583    4633 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716585    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716588    4633 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716590    4633 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716593    4633 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716596    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716598    4633 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716601    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716603    4633 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716608    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716612    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716614    4633 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716617    4633 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716619    4633 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716622    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716625    4633 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716627    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:31.733882 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716630    4633 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716633    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716636    4633 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716639    4633 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716641    4633 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716644    4633 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716646    4633 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716650    4633 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716654    4633 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716657    4633 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716661    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716681    4633 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716686    4633 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716688    4633 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716691    4633 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716694    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716697    4633 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716699    4633 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716702    4633 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:31.734401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716705    4633 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716707    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716710    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716712    4633 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716715    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716718    4633 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716722    4633 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716725    4633 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716727    4633 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716730    4633 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716733    4633 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716735    4633 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716738    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716740    4633 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716743    4633 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716745    4633 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716748    4633 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716750    4633 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716753    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716756    4633 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:31.734941 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716759    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716761    4633 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716764    4633 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716767    4633 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716771    4633 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716773    4633 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716776    4633 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716778    4633 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716781    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716784    4633 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716786    4633 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716789    4633 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716791    4633 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716794    4633 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716796    4633 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716800    4633 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716804    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716808    4633 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716812    4633 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:31.735431 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716815    4633 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716818    4633 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716821    4633 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716823    4633 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716826    4633 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716829    4633 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716832    4633 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716834    4633 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716837    4633 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:31.716839    4633 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.716911    4633 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.717889    4633 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.717942    4633 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.718134    4633 server.go:1014] "Starting client certificate rotation"
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.718265    4633 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.718320    4633 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:55:50 +0000 UTC" deadline="2028-08-10 14:31:14.439871772 +0000 UTC"
Oct 30 21:01:31.736091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.718355    4633 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15593h29m42.721519274s"
Oct 30 21:01:31.736563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.718654    4633 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:01:31.736563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.719933    4633 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:01:31.736563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.722857    4633 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:01:31.736563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.724625    4633 log.go:25] "Validated CRI v1 image API"
Oct 30 21:01:31.736563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.725417    4633 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:01:31.736563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.727113    4633 fs.go:135] Filesystem UUIDs: map[79103255-552b-4ff4-8a93-44b1a133ad37:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7f0ebb78-9780-40a9-9652-bb727666486d:/dev/nvme0n1p3]
Oct 30 21:01:31.736759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.727133    4633 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22907cb2e597dcd95cf4966c60ca85160f6ed2105a67483719654b28242c9c39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22907cb2e597dcd95cf4966c60ca85160f6ed2105a67483719654b28242c9c39/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24752899e8cdadc52d3c7cce6a5ee7774808e50a0bd21fb69c61a6e35ea94cb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24752899e8cdadc52d3c7cce6a5ee7774808e50a0bd21fb69c61a6e35ea94cb9/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c8edaf727a7205225d32400d8cf7494e31c7a005d5ebd0d4e02ead46a22d98c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c8edaf727a7205225d32400d8cf7494e31c7a005d5ebd0d4e02ead46a22d98c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e80823a44905002c59d0b8415f29998656323ce3fab4ce6066e4b9ec4bdb98b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e80823a44905002c59d0b8415f29998656323ce3fab4ce6066e4b9ec4bdb98b/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a4449d529242e23ff53efdd4d7c3db5a0ae65761c4fc988d378586c9ff94a9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a4449d529242e23ff53efdd4d7c3db5a0ae65761c4fc988d378586c9ff94a9f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c7c0bebb2659e387b6cb9ab840be9da3979aa3c86d7597aff85df8d7c963cc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c7c0bebb2659e387b6cb9ab840be9da3979aa3c86d7597aff85df8d7c963cc2/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5779f1266cd56af42347e94914cde8cdfe76b704fd47ef2747dc839222c81a7a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5779f1266cd56af42347e94914cde8cdfe76b704fd47ef2747dc839222c81a7a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61429f4298006dd2eeab41049d2617ac951c50b0a9050cce97dd53062ae593b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61429f4298006dd2eeab41049d2617ac951c50b0a9050cce97dd53062ae593b3/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61860744563f71403a2c860becf15b89d7012cb4ad54301720b902d710a61aa7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61860744563f71403a2c860becf15b89d7012cb4ad54301720b902d710a61aa7/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/691138e98a4e6288cf8a058d70defffe27be7d9148776438227954ae5242b691/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/691138e98a4e6288cf8a058d70defffe27be7d9148776438227954ae5242b691/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98cb2cc90ab939e3a55004b4d377ecabba05d2f7203fc8b4ca4f4d335f7826d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98cb2cc90ab939e3a55004b4d377ecabba05d2f7203fc8b4ca4f4d335f7826d3/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3f28d5aa5d2cd3e609346aa08c4f1d0d96b3ed5c16287742813018be232b4d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3f28d5aa5d2cd3e609346aa08c4f1d0d96b3ed5c16287742813018be232b4d7/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4190d7ce70853a4046e9f978d93b93f5498f879af3183b4d3472127e3cdcb08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4190d7ce70853a4046e9f978d93b93f5498f879af3183b4d3472127e3cdcb08/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e316f7f815709ff003f17b061fdc4173c5d92d2ad392678fb90e22e8291f04b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e316f7f815709ff003f17b061fdc4173c5d92d2ad392678fb90e22e8291f04b7/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4f6375d371e3b287305e9f95ad96b074998c08ffcf7c02fc876c095d6c61236/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4f6375d371e3b287305e9f95ad96b074998c08ffcf7c02fc876c095d6c61236/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb453be1021ff65de923b54bf6ae9d73c7585fd7b9b8a14677ba1be0648b19fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb453be1021ff65de923b54bf6ae9d73c7585fd7b9b8a14677ba1be0648b19fb/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd9449a3541a4af2bc895ccd6e7428e0143d3707af8b781f4a8a633c9275c2d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd9449a3541a4af2bc895ccd6e7428e0143d3707af8b781f4a8a633c9275c2d8/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a6e387b-47f4-4966-b197-e2e73150d514/volumes/kubernetes.io~projected/kube-api-access-ntxws:{mountpoint:/var/lib/kubelet/pods/0a6e387b-47f4-4966-b197-e2e73150d514/volumes/kubernetes.io~projected/kube-api-access-ntxws major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ba46ac0-612c-4964-9454-8ae5e42bd728/volumes/kubernetes.io~projected/kube-api-access-d8xxh:{mountpoint:/var/lib/kubelet/pods/0ba46ac0-612c-4964-9454-8ae5e42bd728/volumes/kubernetes.io~projected/kube-api-access-d8xxh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c66adeb-03be-476e-960e-2544b9605375/volumes/kubernetes.io~projected/kube-api-access-7xzx6:{mountpoint:/var/lib/kubelet/pods/1c66adeb-03be-476e-960e-2544b9605375/volumes/kubernetes.io~projected/kube-api-access-7xzx6 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/205f3154-c424-4dff-9536-cff842b7eac1/volumes/kubernetes.io~projected/kube-api-access-zxwlz:{mountpoint:/var/lib/kubelet/pods/205f3154-c424-4dff-9536-cff842b7eac1/volumes/kubernetes.io~projected/kube-api-access-zxwlz major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/205f3154-c424-4dff-9536-cff842b7eac1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/205f3154-c424-4dff-9536-cff842b7eac1/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/235a4809-212c-45d1-b522-d71ab502e721/volumes/kubernetes.io~projected/kube-api-access-2zmct:{mountpoint:/var/lib/kubelet/pods/235a4809-212c-45d1-b522-d71ab502e721/volumes/kubernetes.io~projected/kube-api-access-2zmct major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/235a4809-212c-45d1-b522-d71ab502e721/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/235a4809-212c-45d1-b522-d71ab502e721/volumes/kubernetes.io~secret/signing-key major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/243d8065-2175-4a20-b229-ebaae8e987fb/volumes/kubernetes.io~projected/kube-api-access-czh7s:{mountpoint:/var/lib/kubelet/pods/243d8065-2175-4a20-b229-ebaae8e987fb/volumes/kubernetes.io~projected/kube-api-access-czh7s major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/243d8065-2175-4a20-b229-ebaae8e987fb/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/243d8065-2175-4a20-b229-ebaae8e987fb/volumes/kubernetes.io~secret/metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2492dc3b-9893-45f5-af15-314312a0a18f/volumes/kubernetes.io~projected/kube-api-access-8mqhx:{mountpoint:/var/lib/kubelet/pods/2492dc3b-9893-45f5-af15-314312a0a18f/volumes/kubernetes.io~projected/kube-api-access-8mqhx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3719ab51-43eb-4857-a169-3711b92d9206/volumes/kubernetes.io~projected/kube-api-access-4v4cf:{mountpoint:/var/lib/kubelet/pods/3719ab51-43eb-4857-a169-3711b92d9206/volumes/kubernetes.io~projected/kube-api-access-4v4cf major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/595d39f4-1364-4e9e-87ae-09120d783bdd/volumes/kubernetes.io~projected/kube-api-access-wn6dv:{mountpoint:/var/lib/kubelet/pods/595d39f4-1364-4e9e-87ae-09120d783bdd/volumes/kubernetes.io~projected/kube-api-access-wn6dv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66f1f33b-3e12-4a49-9ea7-918c9bf78265/volumes/kubernetes.io~projected/kube-api-access-gth79:{mountpoint:/var/lib/kubelet/pods/66f1f33b-3e12-4a49-9ea7-918c9bf78265/volumes/kubernetes.io~projected/kube-api-access-gth79 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~projected/kube-api-access-c7zft:{mountpoint:/var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~projected/kube-api-access-c7zft major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93767113-7f48-4ed1-bb24-d6caaa025dfb/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/93767113-7f48-4ed1-bb24-d6caaa025dfb/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/941b3d3d-7780-4b2d-8ac9-950122fe66a5/volumes/kubernetes.io~projected/kube-api-access-gfpb8:{mountpoint:/var/lib/kubelet/pods/941b3d3d-7780-4b2d-8ac9-950122fe66a5/volumes/kubernetes.io~projected/kube-api-access-gfpb8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6c067ea-93f5-4019-b9e4-100a1c8cb53c/volumes/kubernetes.io~projected/kube-api-access-5c8mg:{mountpoint:/var/lib/kubelet/pods/a6c067ea-93f5-4019-b9e4-100a1c8cb53c/volumes/kubernetes.io~projected/kube-api-access-5c8mg major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a98364d6-40e1-4d54-ba37-84c14da2e2f3/volumes/kubernetes.io~projected/kube-api-access-59bp5:{mountpoint:/var/lib/kubelet/pods/a98364d6-40e1-4d54-ba37-84c14da2e2f3/volumes/kubernetes.io~projected/kube-api-access-59bp5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volumes/kubernetes.io~projected/kube-api-access-67clk:{mountpoint:/var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volumes/kubernetes.io~projected/kube-api-access-67clk major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f71f208a-7acb-47f5-b6c5-eb53347e18f9/volumes/kubernetes.io~projected/kube-api-access-mk79d:{mountpoint:/var/lib/kubelet/pods/f71f208a-7acb-47f5-b6c5-eb53347e18f9/volumes/kubernetes.io~projected/kube-api-access-mk79d major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/267f4af74332d63b130e318fa212e552d89d250899c200d31bcd7bf5bd3252b1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/46f7f4d70bd745e0122ee68ed3e50c181bc9c906745ffb75246aae8357fad84d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/dd0fb50d9c373401c2f405e2598bbb7e7d681ac80dea800f73d23c5688d6146f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/274b878fa153024154ef7c68d34f7de4132efb79c658aa9a05895767f7f9579a/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/016056da912356e3592b688f43adebe346e2cfe2884458c5930c92007ffc5d4d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b4760a404293626871445fa2b4a82557bf72a278435e044f8610603185c01e0f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ab605927bdb5aae040e99e06b7b2db88eac2ce45b4c59f46c7f29923d3c3ac8d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/82ee94b20680929645cf6667e24e9fb2b8939bde37ddadbe14853b77679d5923/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/003e9e14246337be8952ff8fba928f9de3ef9c8b8e68e3fd6eb1ee1697cde007/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b148130d4ef53edb2f8e3e2d466bd62a98a0b04061fa13796987ab6817431ed4/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/8e8b6fbc8c973f974d6f3809762aefecb33beae1a7b9da820f8084fc2158160d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/fa55c030b718962d9122459788ddccfcac2bd2da8bd57f8dbbc12c505604e660/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7ed173d63abb858dc3be34c35dc8dbe6eaf035e64f452ff9be92e4f2759a9f64/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e1914287ae6eece43d8910a483616ab8f4ad64aa014ccb1378d411b5d21a4ecc/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d0c4e017f9e112ad88bc8ba471d0fc66f2e6dadbd013ed44b7ae558b3efe52df/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8fedae7f7fd6933e6fcf182b22e04cc477de1ce6973586a76841387e2ca935bc/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9e552a71c4eff7ddbf57b20d736279f7af91c707b9e55521c51657f5fcd1bf62/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/940be3ec6b325d7261e7a255cdd2feee39a433000b44e7fab5420785db101ea8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/714367a07b633be970fd7a32cf54ce4489410e051c66bc39db64166b434494c0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5cd55f40f6c9517e1b14bd3571f1bb134c5644b2cbd16010f2dfe42a0208aa91/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/494fcb74cb91adb1aece0fc784cedf005fed6c2510862b23eac960b30a3db155/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/f15bd05e5caa4d32545c1a95c5c9e55c418b4657ab7b08ddd1f75d90e899c93b/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/11e0f25573d0ad4cef8d06b909f32068db15556c5d7b41653ff7ebaccf30958a/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/12ba589bb5ce40e61f8de4c44df802c8a31cb673788ecaf333ea41680b78db90/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/9b49496346fc4072c65b2a1fb7f0851f7c65d81ec89141df0d73676dabb0ddf9/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/c65a8807b1f7e9d1f9db85df5b102827796f5be7580c60bd0c5091cc1c2b5e11/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/80b3348abad7ada05069833e8d92ae52e03b4ee550d27ec9257fb02933fc80ee/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/51eef9c521bf3767339571cf13760a717134da9448189d8824856cda440e5c47/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/5415fda11563094eae928e0e7ac2d95fe6dc274fb81f62a7d719d4f11e78c6a5/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/17200d17945c306623232b0dfb94c2d77c4b3b3cecbe7a4ea9268340aace8f9b/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/2fc97553fe13bd1e191cfb2d02f8dd304059c27ab82a9b99dcb63feeff6be285/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/059786414c4985b3cd8600aa41752a4ca0dd1ba1029ad0bf3cb2059e445aa86e/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/5cd1b586c6afc9aab520fe99be009888890c125f210b7d58459ac125de707163/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/1aa0d4e4cea35b5fa2bdef1ee4597e325891023b39d6c2100f10d22bc332ba47/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/7270f9c545c463907eb459671b104d11609126665d097605be33be2242e1dd3c/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/b9d760ea26b460370e23f8d3503729c3f321acd357c9082247e9b4cec041050a/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b09a6b5c751530a63a8f2cf0cc285642781086599f68993deb0dacd3441f53f6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2e429df19f4b29ca8cc64963d4a2cfcc7f200b9d731450a4fb01cf21be95c458/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c302ea0261fed693c3b3c69ceeb4476598d71250540a1e7aa683c5a0718defea/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/dd86d446e1371fc8a73f71fc19ac79ac5d21dad3eb8a5fda8705661bac8d5ba5/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/12660b4cefa1d3eb05682fe45a354082d41f86fe58aa18bc86c8b07cc93b7156/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/68f1afa3def14acfb91cbe0b2e3c6c1a3f2c659c19f2225bcaa48bbdbdc11969/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/471c732f1232216a0a83c7c04c23681af7898af77c16b6efdb8814e8cc314972/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/14c6b3fed60990450a3e763954b2f87f321ad95b9e15a7d10acff01fc2361de9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/592945ccdfa8ea69a173f1cc5c8f85b6c56ce5317246ee4cbc2428260983ec0f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f665381f904faee8166ac107d7d687f1a4fd6b2255e0fd0e58567d7a91d678c0/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.733426    4633 manager.go:217] Machine: {Timestamp:2026-10-30 21:01:31.732963927 +0000 UTC m=+0.082691314 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105066 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29362e93c2570897bfad7ccf1e731a SystemUUID:ec29362e-93c2-5708-97bf-ad7ccf1e731a BootID:a890f721-113a-4c8b-aac7-d1a347654935 Filesystems:[{Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/941b3d3d-7780-4b2d-8ac9-950122fe66a5/volumes/kubernetes.io~projected/kube-api-access-gfpb8 DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98cb2cc90ab939e3a55004b4d377ecabba05d2f7203fc8b4ca4f4d335f7826d3/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24752899e8cdadc52d3c7cce6a5ee7774808e50a0bd21fb69c61a6e35ea94cb9/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/235a4809-212c-45d1-b522-d71ab502e721/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/691138e98a4e6288cf8a058d70defffe27be7d9148776438227954ae5242b691/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2492dc3b-9893-45f5-af15-314312a0a18f/volumes/kubernetes.io~projected/kube-api-access-8mqhx DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a98364d6-40e1-4d54-ba37-84c14da2e2f3/volumes/kubernetes.io~projected/kube-api-access-59bp5 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5779f1266cd56af42347e94914cde8cdfe76b704fd47ef2747dc839222c81a7a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e80823a44905002c59d0b8415f29998656323ce3fab4ce6066e4b9ec4bdb98b/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/243d8065-2175-4a20-b229-ebaae8e987fb/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61429f4298006dd2eeab41049d2617ac951c50b0a9050cce97dd53062ae593b3/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/205f3154-c424-4dff-9536-cff842b7eac1/volumes/kubernetes.io~projected/kube-api-access-zxwlz DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3f28d5aa5d2cd3e609346aa08c4f1d0d96b3ed5c16287742813018be232b4d7/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c66adeb-03be-476e-960e-2544b9605375/volumes/kubernetes.io~projected/kube-api-access-7xzx6 DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/595d39f4-1364-4e9e-87ae-09120d783bdd/volumes/kubernetes.io~projected/kube-api-access-wn6dv DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0ba46ac0-612c-4964-9454-8ae5e42bd728/volumes/kubernetes.io~projected/kube-api-access-d8xxh DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0a6e387b-47f4-4966-b197-e2e73150d514/volumes/kubernetes.io~projected/kube-api-access-ntxws DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~projected/kube-api-access-c7zft DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4f6375d371e3b287305e9f95ad96b074998c08ffcf7c02fc876c095d6c61236/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb453be1021ff65de923b54bf6ae9d73c7585fd7b9b8a14677ba1be0648b19fb/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd9449a3541a4af2bc895ccd6e7428e0143d3707af8b781f4a8a633c9275c2d8/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/243d8065-2175-4a20-b229-ebaae8e987fb/volumes/kubernetes.io~projected/kube-api-access-czh7s DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c7c0bebb2659e387b6cb9ab840be9da3979aa3c86d7597aff85df8d7c963cc2/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66f1f33b-3e12-4a49-9ea7-918c9bf78265/volumes/kubernetes.io~projected/kube-api-access-gth79 DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volumes/kubernetes.io~projected/kube-api-access-67clk DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/205f3154-c424-4dff-9536-cff842b7eac1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e316f7f815709ff003f17b061fdc4173c5d92d2ad392678fb90e22e8291f04b7/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e65d887-3e39-400b-8a4f-7953a5a680e9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/235a4809-212c-45d1-b522-d71ab502e721/volumes/kubernetes.io~projected/kube-api-access-2zmct DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61860744563f71403a2c860becf15b89d7012cb4ad54301720b902d710a61aa7/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22907cb2e597dcd95cf4966c60ca85160f6ed2105a67483719654b28242c9c39/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f71f208a-7acb-47f5-b6c5-eb53347e18f9/volumes/kubernetes.io~projected/kube-api-access-mk79d DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3719ab51-43eb-4857-a169-3711b92d9206/volumes/kubernetes.io~projected/kube-api-access-4v4cf DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6c067ea-93f5-4019-b9e4-100a1c8cb53c/volumes/kubernetes.io~projected/kube-api-access-5c8mg DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4190d7ce70853a4046e9f978d93b93f5498f879af3183b4d3472127e3cdcb08/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/93767113-7f48-4ed1-bb24-d6caaa025dfb/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0130da1-910e-4623-834b-40b494b2309d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a4449d529242e23ff53efdd4d7c3db5a0ae65761c4fc988d378586c9ff94a9f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c8edaf727a7205225d32400d8cf7494e31c7a005d5ebd0d4e02ead46a22d98c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:4c7c0bebb2659e3 MacAddress:72:f0:36:9e:cf:b4 Speed:10000 Mtu:8901} {Name:61860744563f714 MacAddress:8e:86:95:8c:c5:91 Speed:10000 Mtu:8901} {Name:691138e98a4e628 MacAddress:a6:60:60:10:40:7b Speed:10000 Mtu:8901} {Name:98cb2cc90ab939e MacAddress:b6:7c:9c:4f:d5:e2 Speed:10000 Mtu:8901} {Name:a3f28d5aa5d2cd3 MacAddress:e2:ef:1e:db:79:07 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dd:c8:20:23 Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:cc:ba:65:20:56 Speed:0 Mtu:8901} {Name:d4190d7ce70853a MacAddress:da:b9:c6:46:5a:a0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dd:c8:20:23 Speed:0 Mtu:9001} {Name:fd9449a3541a4af MacAddress:fe:5f:33:c3:e2:6f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:06:89:08:18:e9:1e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:6a:09:36:2c:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.733706    4633 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.733756    4633 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734017    4633 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734037    4633 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-12.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734182    4633 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734190    4633 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734201    4633 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734214    4633 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734296    4633 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734464    4633 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734573    4633 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734611    4633 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734621    4633 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734633    4633 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734642    4633 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.734650    4633 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.735338    4633 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.735347    4633 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:01:31.736903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.735861    4633 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:01:31.739766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.739317    4633 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:01:31.739766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.739337    4633 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:01:31.739766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.739497    4633 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:01:31.739939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.739865    4633 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:01:31.739939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.739909    4633 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:01:31.740103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740077    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:01:31.740148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740133    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:01:31.740190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740155    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:01:31.740190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740167    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:01:31.740260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740222    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:01:31.740260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740237    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:01:31.740509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740496    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:01:31.740579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740525    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:01:31.740579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740542    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:01:31.740579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740553    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:01:31.740579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740567    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:01:31.740780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740585    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:01:31.740780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740626    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:01:31.740780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.740636    4633 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:01:31.744162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.744144    4633 server.go:1263] "Started kubelet"
Oct 30 21:01:31.744455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.744368    4633 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:01:31.744531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.744407    4633 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:01:31.744578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.744568    4633 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:01:31.744730 ip-10-0-140-12 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:01:31.745956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.744898    4633 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:01:31.746894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.746873    4633 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.748153    4633 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.748395    4633 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.748487    4633 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.748705    4633 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:55:53 +0000 UTC" deadline="2028-07-13 02:16:47.820790612 +0000 UTC"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.748767    4633 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14909h15m16.0720272s"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.748960    4633 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.748971    4633 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.749839    4633 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.750500    4633 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753233    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c66adeb-03be-476e-960e-2544b9605375" volumeName="kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6" seLinuxMountContext=""
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753311    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3719ab51-43eb-4857-a169-3711b92d9206" volumeName="kubernetes.io/projected/3719ab51-43eb-4857-a169-3711b92d9206-kube-api-access-4v4cf" seLinuxMountContext=""
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753345    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e65d887-3e39-400b-8a4f-7953a5a680e9" volumeName="kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-tuned" seLinuxMountContext=""
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753359    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="941b3d3d-7780-4b2d-8ac9-950122fe66a5" volumeName="kubernetes.io/projected/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-api-access-gfpb8" seLinuxMountContext=""
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753384    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0130da1-910e-4623-834b-40b494b2309d" volumeName="kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753402    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="243d8065-2175-4a20-b229-ebaae8e987fb" volumeName="kubernetes.io/empty-dir/243d8065-2175-4a20-b229-ebaae8e987fb-tmp-dir" seLinuxMountContext=""
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753417    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="243d8065-2175-4a20-b229-ebaae8e987fb" volumeName="kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls" seLinuxMountContext=""
Oct 30 21:01:31.753422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753433    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2492dc3b-9893-45f5-af15-314312a0a18f" volumeName="kubernetes.io/empty-dir/2492dc3b-9893-45f5-af15-314312a0a18f-tmp-dir" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753505    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="235a4809-212c-45d1-b522-d71ab502e721" volumeName="kubernetes.io/secret/235a4809-212c-45d1-b522-d71ab502e721-signing-key" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753521    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a6e387b-47f4-4966-b197-e2e73150d514" volumeName="kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753550    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a6e387b-47f4-4966-b197-e2e73150d514" volumeName="kubernetes.io/projected/0a6e387b-47f4-4966-b197-e2e73150d514-kube-api-access-ntxws" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753565    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="941b3d3d-7780-4b2d-8ac9-950122fe66a5" volumeName="kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753584    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ba46ac0-612c-4964-9454-8ae5e42bd728" volumeName="kubernetes.io/projected/0ba46ac0-612c-4964-9454-8ae5e42bd728-kube-api-access-d8xxh" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753597    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c66adeb-03be-476e-960e-2544b9605375" volumeName="kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753631    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="205f3154-c424-4dff-9536-cff842b7eac1" volumeName="kubernetes.io/secret/205f3154-c424-4dff-9536-cff842b7eac1-cert" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753647    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="595d39f4-1364-4e9e-87ae-09120d783bdd" volumeName="kubernetes.io/projected/595d39f4-1364-4e9e-87ae-09120d783bdd-kube-api-access-wn6dv" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753711    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0130da1-910e-4623-834b-40b494b2309d" volumeName="kubernetes.io/secret/e0130da1-910e-4623-834b-40b494b2309d-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753728    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a6e387b-47f4-4966-b197-e2e73150d514" volumeName="kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753762    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66f1f33b-3e12-4a49-9ea7-918c9bf78265" volumeName="kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753801    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e65d887-3e39-400b-8a4f-7953a5a680e9" volumeName="kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-tmp" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753836    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e65d887-3e39-400b-8a4f-7953a5a680e9" volumeName="kubernetes.io/projected/7e65d887-3e39-400b-8a4f-7953a5a680e9-kube-api-access-c7zft" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753853    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93767113-7f48-4ed1-bb24-d6caaa025dfb" volumeName="kubernetes.io/secret/93767113-7f48-4ed1-bb24-d6caaa025dfb-agent-certs" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753876    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0130da1-910e-4623-834b-40b494b2309d" volumeName="kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-config" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753890    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="595d39f4-1364-4e9e-87ae-09120d783bdd" volumeName="kubernetes.io/configmap/595d39f4-1364-4e9e-87ae-09120d783bdd-serviceca" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753917    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66f1f33b-3e12-4a49-9ea7-918c9bf78265" volumeName="kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753931    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66f1f33b-3e12-4a49-9ea7-918c9bf78265" volumeName="kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753959    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0130da1-910e-4623-834b-40b494b2309d" volumeName="kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-env-overrides" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.753985    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="235a4809-212c-45d1-b522-d71ab502e721" volumeName="kubernetes.io/projected/235a4809-212c-45d1-b522-d71ab502e721-kube-api-access-2zmct" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754024    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="243d8065-2175-4a20-b229-ebaae8e987fb" volumeName="kubernetes.io/configmap/243d8065-2175-4a20-b229-ebaae8e987fb-config-volume" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754038    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c66adeb-03be-476e-960e-2544b9605375" volumeName="kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754056    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c66adeb-03be-476e-960e-2544b9605375" volumeName="kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754068    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="205f3154-c424-4dff-9536-cff842b7eac1" volumeName="kubernetes.io/projected/205f3154-c424-4dff-9536-cff842b7eac1-kube-api-access-zxwlz" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754102    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="235a4809-212c-45d1-b522-d71ab502e721" volumeName="kubernetes.io/configmap/235a4809-212c-45d1-b522-d71ab502e721-signing-cabundle" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754152    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6b17092-1d1d-40e0-b85f-28f05ab9f5f6" volumeName="kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754176    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0130da1-910e-4623-834b-40b494b2309d" volumeName="kubernetes.io/projected/e0130da1-910e-4623-834b-40b494b2309d-kube-api-access-67clk" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754194    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="243d8065-2175-4a20-b229-ebaae8e987fb" volumeName="kubernetes.io/projected/243d8065-2175-4a20-b229-ebaae8e987fb-kube-api-access-czh7s" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754207    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2492dc3b-9893-45f5-af15-314312a0a18f" volumeName="kubernetes.io/projected/2492dc3b-9893-45f5-af15-314312a0a18f-kube-api-access-8mqhx" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754219    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66f1f33b-3e12-4a49-9ea7-918c9bf78265" volumeName="kubernetes.io/projected/66f1f33b-3e12-4a49-9ea7-918c9bf78265-kube-api-access-gth79" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754246    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="941b3d3d-7780-4b2d-8ac9-950122fe66a5" volumeName="kubernetes.io/empty-dir/941b3d3d-7780-4b2d-8ac9-950122fe66a5-data-volume" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754261    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f71f208a-7acb-47f5-b6c5-eb53347e18f9" volumeName="kubernetes.io/configmap/f71f208a-7acb-47f5-b6c5-eb53347e18f9-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754300    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93767113-7f48-4ed1-bb24-d6caaa025dfb" volumeName="kubernetes.io/configmap/93767113-7f48-4ed1-bb24-d6caaa025dfb-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:01:31.754319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754314    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6c067ea-93f5-4019-b9e4-100a1c8cb53c" volumeName="kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg" seLinuxMountContext=""
Oct 30 21:01:31.755773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754327    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a98364d6-40e1-4d54-ba37-84c14da2e2f3" volumeName="kubernetes.io/projected/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kube-api-access-59bp5" seLinuxMountContext=""
Oct 30 21:01:31.755773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754392    4633 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f71f208a-7acb-47f5-b6c5-eb53347e18f9" volumeName="kubernetes.io/projected/f71f208a-7acb-47f5-b6c5-eb53347e18f9-kube-api-access-mk79d" seLinuxMountContext=""
Oct 30 21:01:31.755773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754404    4633 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:01:31.755773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.754412    4633 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:01:31.755773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.755352    4633 factory.go:55] Registering systemd factory
Oct 30 21:01:31.755773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.755370    4633 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:01:31.755773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.755714    4633 factory.go:153] Registering CRI-O factory
Oct 30 21:01:31.755773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.755752    4633 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:01:31.756131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.755825    4633 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:01:31.756131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.755845    4633 factory.go:103] Registering Raw factory
Oct 30 21:01:31.756131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.755866    4633 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:01:31.756460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.756440    4633 manager.go:319] Starting recovery of all containers
Oct 30 21:01:31.758611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.757499    4633 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:01:31.758611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.757515    4633 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:01:31.769526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.769496    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:01:31.769773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.769752    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:01:31.781236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.781187    4633 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:01:31.782572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.782551    4633 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:01:31.782572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.782571    4633 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:01:31.782760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.782591    4633 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:01:31.782760 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:31.782627    4633 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:01:31.789569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.787354    4633 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:01:31.803721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.803687    4633 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="56832e56cfb94ba29c640a86ec5b8dfb82eeacc9033b1eb3b145a3d206a96f15" exitCode=0
Oct 30 21:01:31.803721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.803716    4633 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="4ffb9b36d18162a127d2d7df056562b547abd70c285adc3323b884d519462f29" exitCode=0
Oct 30 21:01:31.803721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.803723    4633 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="b534b69486be80baacc417c817ad44a599836e6e4389a72e5c7640f7d966149f" exitCode=0
Oct 30 21:01:31.803977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.803738    4633 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="d7a1fedabcc559c71826fc3a4aa60b9ff5780c6d383fd45db347348cf690cb26" exitCode=0
Oct 30 21:01:31.803977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.803745    4633 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="240a49a5729c1f0aba876c6f66fda59cd38e75411f4783ebf9d11cfec86c5297" exitCode=0
Oct 30 21:01:31.803977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.803752    4633 generic.go:356] "Generic (PLEG): container finished" podID="66f1f33b-3e12-4a49-9ea7-918c9bf78265" containerID="ac09516245d8a4b8fe9de4d7a84d7ebd93bca51947b8a4ff9ffdee058d56643a" exitCode=0
Oct 30 21:01:31.817549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.817511    4633 generic.go:356] "Generic (PLEG): container finished" podID="c57946f0cf74e97be779f3304bd4cc78" containerID="471d7b98c95708bcf37f00231fa0320296d3bb927a3272a7219d1003d79abd55" exitCode=0
Oct 30 21:01:31.825102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.825079    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:01:31.825436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.825412    4633 generic.go:356] "Generic (PLEG): container finished" podID="e0130da1-910e-4623-834b-40b494b2309d" containerID="ce4ebfa129700630d7d77211f1270e584a58f9858d09ace1fbd8c917292329bf" exitCode=1
Oct 30 21:01:31.833931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.833916    4633 manager.go:324] Recovery completed
Oct 30 21:01:31.851789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851767    4633 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:01:31.851789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851783    4633 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:01:31.851935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851802    4633 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:01:31.851935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851927    4633 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:01:31.851988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851937    4633 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:01:31.851988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851954    4633 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:01:31.851988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851959    4633 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:01:31.851988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851964    4633 policy_none.go:50] "Start"
Oct 30 21:01:31.851988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851972    4633 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:01:31.851988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.851981    4633 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:01:31.852136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.852100    4633 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:01:31.852136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.852107    4633 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:01:31.852136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.852113    4633 policy_none.go:44] "Start"
Oct 30 21:01:31.856113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.856085    4633 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:01:31.860235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.860216    4633 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:01:31.860330 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:31.860249    4633 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:01:31.860330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.860262    4633 server.go:87] "Starting device plugin registration server"
Oct 30 21:01:31.861307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.861287    4633 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:01:31.861397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.861306    4633 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:01:31.861449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.861425    4633 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:01:31.861552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.861513    4633 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:01:31.861651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.861564    4633 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:01:31.861651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.861573    4633 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:01:31.861788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.861632    4633 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:01:31.861550266Z","UUID":"7a84bd9d-c71a-4fd5-9b79-445c280ff306","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:01:31.863272 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:31.863183    4633 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:01:31.863791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.863764    4633 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:01:31.863914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.863808    4633 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:01:31.882757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.882710    4633 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"]
Oct 30 21:01:31.883313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.883261    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"070c8221ace6d9936bb76d676bde1ea5","Type":"ContainerStarted","Data":"0f57e52b348d2ef635c82ee7cea6b6ed2dc97b0ec24614e9d2ce64b6ed639983"}
Oct 30 21:01:31.883400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.883323    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"070c8221ace6d9936bb76d676bde1ea5","Type":"ContainerStarted","Data":"22907cb2e597dcd95cf4966c60ca85160f6ed2105a67483719654b28242c9c39"}
Oct 30 21:01:31.883400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.883369    4633 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.883400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.883391    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"c57946f0cf74e97be779f3304bd4cc78","Type":"ContainerStarted","Data":"0a48dacfd4fd3057d187f044e09e3e7c594f10b4a3084701d5537704d2f0cacc"}
Oct 30 21:01:31.883530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.883412    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"c57946f0cf74e97be779f3304bd4cc78","Type":"ContainerDied","Data":"471d7b98c95708bcf37f00231fa0320296d3bb927a3272a7219d1003d79abd55"}
Oct 30 21:01:31.883530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.883433    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal" event={"ID":"c57946f0cf74e97be779f3304bd4cc78","Type":"ContainerStarted","Data":"e316f7f815709ff003f17b061fdc4173c5d92d2ad392678fb90e22e8291f04b7"}
Oct 30 21:01:31.883636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.883623    4633 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.894828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.894809    4633 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:01:31.894897 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:31.894862    4633 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.895265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.895247    4633 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:01:31.895343 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:31.895291    4633 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.958685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.958642    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/070c8221ace6d9936bb76d676bde1ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"070c8221ace6d9936bb76d676bde1ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.958824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.958708    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.958824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.958731    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.961746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.961724    4633 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:01:31.963174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.963158    4633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:01:31.963273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.963196    4633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:01:31.963273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.963211    4633 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-12.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:01:31.963273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.963257    4633 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.973540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.973523    4633 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:31.973589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:31.973585    4633 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.059649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.059557    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/070c8221ace6d9936bb76d676bde1ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"070c8221ace6d9936bb76d676bde1ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.059649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.059585    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/070c8221ace6d9936bb76d676bde1ea5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"070c8221ace6d9936bb76d676bde1ea5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.059649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.059640    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.059902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.059661    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.059902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.059723    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.059902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.059761    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c57946f0cf74e97be779f3304bd4cc78-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" (UID: \"c57946f0cf74e97be779f3304bd4cc78\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.735005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.734924    4633 apiserver.go:52] "Watching apiserver"
Oct 30 21:01:32.745140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.745115    4633 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:01:32.745561 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.745543    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-r7gpg","openshift-image-registry/node-ca-wrffd","openshift-ovn-kubernetes/ovnkube-node-4cz8g","kube-system/global-pull-secret-syncer-lbvj4","openshift-multus/network-metrics-daemon-gbg4d","openshift-network-diagnostics/network-check-target-kwb4j","openshift-network-operator/iptables-alerter-vmp52","openshift-service-ca/service-ca-5fffc97bdf-d2r2t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv","openshift-multus/multus-additional-cni-plugins-mfwjh","openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np","kube-system/konnectivity-agent-r2kvg","openshift-cluster-node-tuning-operator/tuned-9hs5s","openshift-ingress-canary/ingress-canary-jq4p6","openshift-insights/insights-runtime-extractor-b7hnm","kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal","openshift-dns/dns-default-5l9cm","openshift-insights/periodic-gathering-rtfgg-7ntt6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal","openshift-multus/multus-5sxfl"]
Oct 30 21:01:32.747612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.747590    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:32.749941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.749909    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v7745\""
Oct 30 21:01:32.749941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.749909    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g68zh\""
Oct 30 21:01:32.750929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.750909    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x8hl8\""
Oct 30 21:01:32.751489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.751470    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.751589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.751569    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.751589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.751472    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6bj8t\""
Oct 30 21:01:32.751717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.751608    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:01:32.751717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.751471    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.751945 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.751753    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.752141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.752116    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:01:32.752883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.752858    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:01:32.752883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.752876    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qbtrh\""
Oct 30 21:01:32.753942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.753920    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:01:32.754126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.754111    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.754234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.754222    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.754451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.754436    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.754585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.754572    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:01:32.754800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.754786    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:01:32.754863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.754578    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.755486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755466    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:01:32.755566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755499    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:01:32.755633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755595    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wx578\""
Oct 30 21:01:32.755633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755609    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qd7pl\""
Oct 30 21:01:32.755774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755640    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.755774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755660    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:01:32.755774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755739    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.755913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755826    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:01:32.755913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755892    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfb4q\""
Oct 30 21:01:32.756010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755976    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:01:32.756010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755997    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:01:32.756137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.756068    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6m5dr\""
Oct 30 21:01:32.756137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.755979    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.756224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.756193    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.756276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.756236    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mwc48\""
Oct 30 21:01:32.756367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.756351    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.756423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.756386    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.756818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.756800    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.757023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757007    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.757352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757336    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.757722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757705    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.757827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757810    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:01:32.757879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757851    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-99mk5\""
Oct 30 21:01:32.757939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757879    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:01:32.757939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757895    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:01:32.757939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757895    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hspc2\""
Oct 30 21:01:32.758074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.757978    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.758074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.758036    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s7wtw\""
Oct 30 21:01:32.758393 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.758375    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:01:32.758692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.758648    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:01:32.758821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.758803    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cp8br\""
Oct 30 21:01:32.759011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.758998    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:01:32.760205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.760192    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.760706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.760690    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lw2cs\""
Oct 30 21:01:32.760834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.760819    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:01:32.760906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.760844    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.760906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.760872    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.760906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.760878    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpqzq\""
Oct 30 21:01:32.760906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.760896    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.761159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.761147    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:01:32.761917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.761897    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:01:32.761992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.761918    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:01:32.761992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.761986    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k54sx\""
Oct 30 21:01:32.762086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.761994    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:01:32.762086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762051    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dcwvc\""
Oct 30 21:01:32.762086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762071    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:01:32.762181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762120    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:01:32.762181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762138    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-systemd\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.762181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762177    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.762304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762213    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.762304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762241    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-script-lib\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.762304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762276    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f71f208a-7acb-47f5-b6c5-eb53347e18f9-host-slash\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:01:32.762410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762311    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-socket-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.762410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762342    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-systemd-units\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.762410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762380    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-var-lib-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.762546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762439    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2492dc3b-9893-45f5-af15-314312a0a18f-hosts-file\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:01:32.762546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762472    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-dbus\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:32.762546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762498    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-script-lib\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.762546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762499    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-system-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.762779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762563    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-conf-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.762779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762581    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-sys-fs\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.762779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762600    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.762779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762632    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-etc-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.762779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762692    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-ovn\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.762779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762729    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.762779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762743    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysconfig\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.762779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762765    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-kubelet\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762790    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-hostroot\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762820    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bp5\" (UniqueName: \"kubernetes.io/projected/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kube-api-access-59bp5\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762852    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-node-log\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762913    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-modprobe-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762937    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-var-lib-kubelet\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762962    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-cni-binary-copy\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.762992    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763003    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67clk\" (UniqueName: \"kubernetes.io/projected/e0130da1-910e-4623-834b-40b494b2309d-kube-api-access-67clk\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763030    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-kubelet-config\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763051    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.763121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763078    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-run\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763190    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-netns\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763235    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763266    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gth79\" (UniqueName: \"kubernetes.io/projected/66f1f33b-3e12-4a49-9ea7-918c9bf78265-kube-api-access-gth79\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763296    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6dv\" (UniqueName: \"kubernetes.io/projected/595d39f4-1364-4e9e-87ae-09120d783bdd-kube-api-access-wn6dv\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763317    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-cni-binary-copy\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763318    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-systemd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763364    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-netd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763396    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763530    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-sys\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.763586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763583    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-lib-modules\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763618    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-original-pull-secret\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763618    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk79d\" (UniqueName: \"kubernetes.io/projected/f71f208a-7acb-47f5-b6c5-eb53347e18f9-kube-api-access-mk79d\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763663    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-k8s-cni-cncf-io\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763713    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-multus\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763756    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cnibin\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763784    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-bin\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763801    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-kubernetes\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763824    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-tuned\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763862    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-netns\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763898    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763922    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-tuned\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763938    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-slash\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763969    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-bin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.763994    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.764050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764026    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764059    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764094    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xxh\" (UniqueName: \"kubernetes.io/projected/0ba46ac0-612c-4964-9454-8ae5e42bd728-kube-api-access-d8xxh\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764109    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-conf\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764133    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-host\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764146    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764156    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7zft\" (UniqueName: \"kubernetes.io/projected/7e65d887-3e39-400b-8a4f-7953a5a680e9-kube-api-access-c7zft\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764171    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764186    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2492dc3b-9893-45f5-af15-314312a0a18f-tmp-dir\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764197    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cni-binary-copy\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764235    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-env-overrides\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764271    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2492dc3b-9893-45f5-af15-314312a0a18f-tmp-dir\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764368    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764421    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-socket-dir-parent\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764498    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-env-overrides\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.764631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764601    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-multus-certs\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764650    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-registration-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764706    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/595d39f4-1364-4e9e-87ae-09120d783bdd-host\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764759    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-config\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764778    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764793    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f71f208a-7acb-47f5-b6c5-eb53347e18f9-iptables-alerter-script\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764816    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-cnibin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764927    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-etc-kubernetes\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764961    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/595d39f4-1364-4e9e-87ae-09120d783bdd-serviceca\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.764980    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e0130da1-910e-4623-834b-40b494b2309d-ovnkube-config\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765013    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-multus-daemon-config\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765044    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-tmp\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765073    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-kubelet\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.765098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765098    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-system-cni-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765121    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mqhx\" (UniqueName: \"kubernetes.io/projected/2492dc3b-9893-45f5-af15-314312a0a18f-kube-api-access-8mqhx\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765150    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e65d887-3e39-400b-8a4f-7953a5a680e9-tmp\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765162    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0130da1-910e-4623-834b-40b494b2309d-ovn-node-metrics-cert\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765225    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxws\" (UniqueName: \"kubernetes.io/projected/0a6e387b-47f4-4966-b197-e2e73150d514-kube-api-access-ntxws\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765292    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f71f208a-7acb-47f5-b6c5-eb53347e18f9-iptables-alerter-script\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765299    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0a6e387b-47f4-4966-b197-e2e73150d514-multus-daemon-config\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765351    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-device-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765363    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0130da1-910e-4623-834b-40b494b2309d-ovn-node-metrics-cert\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.765572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765379    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-log-socket\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.765975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.765960    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:01:32.766121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.766103    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/595d39f4-1364-4e9e-87ae-09120d783bdd-serviceca\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:01:32.769244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.769223    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bp5\" (UniqueName: \"kubernetes.io/projected/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kube-api-access-59bp5\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.769524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.769506    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6dv\" (UniqueName: \"kubernetes.io/projected/595d39f4-1364-4e9e-87ae-09120d783bdd-kube-api-access-wn6dv\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:01:32.769524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.769516    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk79d\" (UniqueName: \"kubernetes.io/projected/f71f208a-7acb-47f5-b6c5-eb53347e18f9-kube-api-access-mk79d\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:01:32.769862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.769847    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67clk\" (UniqueName: \"kubernetes.io/projected/e0130da1-910e-4623-834b-40b494b2309d-kube-api-access-67clk\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.769898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.769863    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gth79\" (UniqueName: \"kubernetes.io/projected/66f1f33b-3e12-4a49-9ea7-918c9bf78265-kube-api-access-gth79\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.770378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.770363    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xxh\" (UniqueName: \"kubernetes.io/projected/0ba46ac0-612c-4964-9454-8ae5e42bd728-kube-api-access-d8xxh\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:32.770515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.770497    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7zft\" (UniqueName: \"kubernetes.io/projected/7e65d887-3e39-400b-8a4f-7953a5a680e9-kube-api-access-c7zft\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.770964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.770948    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8mg\" (UniqueName: \"kubernetes.io/projected/a6c067ea-93f5-4019-b9e4-100a1c8cb53c-kube-api-access-5c8mg\") pod \"network-check-target-kwb4j\" (UID: \"a6c067ea-93f5-4019-b9e4-100a1c8cb53c\") " pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:32.771227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.771212    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mqhx\" (UniqueName: \"kubernetes.io/projected/2492dc3b-9893-45f5-af15-314312a0a18f-kube-api-access-8mqhx\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:01:32.771421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.771402    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxws\" (UniqueName: \"kubernetes.io/projected/0a6e387b-47f4-4966-b197-e2e73150d514-kube-api-access-ntxws\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.830850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.830821    4633 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.831002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.830986    4633 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.842192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.842167    4633 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:01:32.842331 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:32.842202    4633 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.842650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.842636    4633 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:01:32.842718 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:32.842683    4633 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-12.us-east-2.compute.internal"
Oct 30 21:01:32.852472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.852392    4633 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:01:32.866110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866082    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmct\" (UniqueName: \"kubernetes.io/projected/235a4809-212c-45d1-b522-d71ab502e721-kube-api-access-2zmct\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:32.866110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866110    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.866285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866129    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-conf\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.866285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866143    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-host\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.866285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866158    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866173    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:32.866285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866203    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.866285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866234    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-host\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.866285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866255    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866285 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:32.866258    4633 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866297    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-socket-dir-parent\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866303    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-conf\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:32.866341    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:33.366325668 +0000 UTC m=+1.716053065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : secret "metrics-daemon-secret" not found
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866351    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-socket-dir-parent\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866355    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-multus-certs\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866388    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-multus-certs\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866387    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-registration-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866410    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243d8065-2175-4a20-b229-ebaae8e987fb-tmp-dir\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866426    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh7s\" (UniqueName: \"kubernetes.io/projected/243d8065-2175-4a20-b229-ebaae8e987fb-kube-api-access-czh7s\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866432    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-registration-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866443    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/595d39f4-1364-4e9e-87ae-09120d783bdd-host\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866464    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-cnibin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866478    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-etc-kubernetes\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866495    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/243d8065-2175-4a20-b229-ebaae8e987fb-config-volume\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866509    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/595d39f4-1364-4e9e-87ae-09120d783bdd-host\") pod \"node-ca-wrffd\" (UID: \"595d39f4-1364-4e9e-87ae-09120d783bdd\") " pod="openshift-image-registry/node-ca-wrffd"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866519    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-kubelet\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866531    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243d8065-2175-4a20-b229-ebaae8e987fb-tmp-dir\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866540    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-cnibin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866544    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-system-cni-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866561    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-etc-kubernetes\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866578    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-system-cni-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866584    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-kubelet\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.866623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866614    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-device-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866639    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzx6\" (UniqueName: \"kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866660    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-log-socket\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866709    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/941b3d3d-7780-4b2d-8ac9-950122fe66a5-data-volume\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866729    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-log-socket\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866730    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866762    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-systemd\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866786    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866811    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866710    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-device-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866836    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205f3154-c424-4dff-9536-cff842b7eac1-cert\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866875    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/243d8065-2175-4a20-b229-ebaae8e987fb-config-volume\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866884    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866904    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866926    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d8065-2175-4a20-b229-ebaae8e987fb-metrics-tls\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866931    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-systemd\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.866966    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867032    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f71f208a-7acb-47f5-b6c5-eb53347e18f9-host-slash\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867036    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/941b3d3d-7780-4b2d-8ac9-950122fe66a5-data-volume\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867037    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867082    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205f3154-c424-4dff-9536-cff842b7eac1-cert\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867083    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-socket-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867113    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwlz\" (UniqueName: \"kubernetes.io/projected/205f3154-c424-4dff-9536-cff842b7eac1-kube-api-access-zxwlz\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867092    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f71f208a-7acb-47f5-b6c5-eb53347e18f9-host-slash\") pod \"iptables-alerter-vmp52\" (UID: \"f71f208a-7acb-47f5-b6c5-eb53347e18f9\") " pod="openshift-network-operator/iptables-alerter-vmp52"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867126    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-socket-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867158    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-systemd-units\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867223    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-var-lib-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867231    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-systemd-units\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867260    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2492dc3b-9893-45f5-af15-314312a0a18f-hosts-file\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867286    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2492dc3b-9893-45f5-af15-314312a0a18f-hosts-file\") pod \"node-resolver-r7gpg\" (UID: \"2492dc3b-9893-45f5-af15-314312a0a18f\") " pod="openshift-dns/node-resolver-r7gpg"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867304    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-var-lib-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867328    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-dbus\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867361    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-system-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867404    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-dbus\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867431    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-conf-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867432    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-system-cni-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867506    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-sys-fs\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867545    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-multus-conf-dir\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867567    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-etc-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867605    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-etc-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867613    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93767113-7f48-4ed1-bb24-d6caaa025dfb-agent-certs\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:01:32.867619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867643    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-sys-fs\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867694    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867788    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/941b3d3d-7780-4b2d-8ac9-950122fe66a5-crio-socket\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867806    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-ovn\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867836    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867859    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysconfig\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867882    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-kubelet\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867893    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/93767113-7f48-4ed1-bb24-d6caaa025dfb-agent-certs\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867892    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-ovn\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867927    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-hostroot\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867943    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-ovn-kubernetes\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867945    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867964    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysconfig\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867963    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-kubelet\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.867976    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-hostroot\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868004    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93767113-7f48-4ed1-bb24-d6caaa025dfb-konnectivity-ca\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868030    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868046    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-node-log\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868067    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-modprobe-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868090    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-var-lib-kubelet\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868091    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-node-log\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868133    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-var-lib-kubelet\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868169    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v4cf\" (UniqueName: \"kubernetes.io/projected/3719ab51-43eb-4857-a169-3711b92d9206-kube-api-access-4v4cf\") pod \"migrator-66df5d496b-l56np\" (UID: \"3719ab51-43eb-4857-a169-3711b92d9206\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868179    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/93767113-7f48-4ed1-bb24-d6caaa025dfb-konnectivity-ca\") pod \"konnectivity-agent-r2kvg\" (UID: \"93767113-7f48-4ed1-bb24-d6caaa025dfb\") " pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868186    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-modprobe-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868214    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868246    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-kubelet-config\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868267    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868273    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868290    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c6b17092-1d1d-40e0-b85f-28f05ab9f5f6-kubelet-config\") pod \"global-pull-secret-syncer-lbvj4\" (UID: \"c6b17092-1d1d-40e0-b85f-28f05ab9f5f6\") " pod="kube-system/global-pull-secret-syncer-lbvj4"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868291    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-run\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868315    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-netns\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868339    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868348    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-run\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868362    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-systemd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868366    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-run-netns\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868379    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-netd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868381    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-sysctl-d\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868394    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-sys\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868411    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-netd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868392    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-openvswitch\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868422    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-sys\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868414    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-lib-modules\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868393    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-run-systemd\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868450    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-k8s-cni-cncf-io\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868469    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-multus\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868485    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cnibin\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868505    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-lib-modules\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868519    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-bin\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868535    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-k8s-cni-cncf-io\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868542    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-kubernetes\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868550    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f1f33b-3e12-4a49-9ea7-918c9bf78265-cnibin\") pod \"multus-additional-cni-plugins-mfwjh\" (UID: \"66f1f33b-3e12-4a49-9ea7-918c9bf78265\") " pod="openshift-multus/multus-additional-cni-plugins-mfwjh"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868540    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-multus\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868578    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-netns\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868601    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/235a4809-212c-45d1-b522-d71ab502e721-signing-cabundle\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868612    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e65d887-3e39-400b-8a4f-7953a5a680e9-etc-kubernetes\") pod \"tuned-9hs5s\" (UID: \"7e65d887-3e39-400b-8a4f-7953a5a680e9\") " pod="openshift-cluster-node-tuning-operator/tuned-9hs5s"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868616    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-slash\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868637    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-slash\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868637    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-bin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868599    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0130da1-910e-4623-834b-40b494b2309d-host-cni-bin\") pod \"ovnkube-node-4cz8g\" (UID: \"e0130da1-910e-4623-834b-40b494b2309d\") " pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868662    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-var-lib-cni-bin\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868664    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868705    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a6e387b-47f4-4966-b197-e2e73150d514-host-run-netns\") pod \"multus-5sxfl\" (UID: \"0a6e387b-47f4-4966-b197-e2e73150d514\") " pod="openshift-multus/multus-5sxfl"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868717    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a98364d6-40e1-4d54-ba37-84c14da2e2f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6zscv\" (UID: \"a98364d6-40e1-4d54-ba37-84c14da2e2f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6zscv"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868709    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868748    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpb8\" (UniqueName: \"kubernetes.io/projected/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-api-access-gfpb8\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868767    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/235a4809-212c-45d1-b522-d71ab502e721-signing-key\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868848    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/235a4809-212c-45d1-b522-d71ab502e721-signing-cabundle\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868935    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/235a4809-212c-45d1-b522-d71ab502e721-signing-key\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:32.869368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.868993    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:32.874854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.874837    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmct\" (UniqueName: \"kubernetes.io/projected/235a4809-212c-45d1-b522-d71ab502e721-kube-api-access-2zmct\") pod \"service-ca-5fffc97bdf-d2r2t\" (UID: \"235a4809-212c-45d1-b522-d71ab502e721\") " pod="openshift-service-ca/service-ca-5fffc97bdf-d2r2t"
Oct 30 21:01:32.875305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.875280    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v4cf\" (UniqueName: \"kubernetes.io/projected/3719ab51-43eb-4857-a169-3711b92d9206-kube-api-access-4v4cf\") pod \"migrator-66df5d496b-l56np\" (UID: \"3719ab51-43eb-4857-a169-3711b92d9206\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-l56np"
Oct 30 21:01:32.875398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.875379    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh7s\" (UniqueName: \"kubernetes.io/projected/243d8065-2175-4a20-b229-ebaae8e987fb-kube-api-access-czh7s\") pod \"dns-default-5l9cm\" (UID: \"243d8065-2175-4a20-b229-ebaae8e987fb\") " pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:32.875431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.875379    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpb8\" (UniqueName: \"kubernetes.io/projected/941b3d3d-7780-4b2d-8ac9-950122fe66a5-kube-api-access-gfpb8\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.875490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.875471    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwlz\" (UniqueName: \"kubernetes.io/projected/205f3154-c424-4dff-9536-cff842b7eac1-kube-api-access-zxwlz\") pod \"ingress-canary-jq4p6\" (UID: \"205f3154-c424-4dff-9536-cff842b7eac1\") " pod="openshift-ingress-canary/ingress-canary-jq4p6"
Oct 30 21:01:32.875527 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.875468    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzx6\" (UniqueName: \"kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6\") pod \"periodic-gathering-rtfgg-7ntt6\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") " pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:01:32.969550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.969520    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/941b3d3d-7780-4b2d-8ac9-950122fe66a5-crio-socket\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.969715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.969556    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.969715 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:32.969650    4633 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:32.969715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:32.969658    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/941b3d3d-7780-4b2d-8ac9-950122fe66a5-crio-socket\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:32.969826 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:32.969735    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:33.469720578 +0000 UTC m=+1.819447965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:33.373490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:33.373452    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:33.373639 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:33.373576    4633 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:01:33.373639 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:33.373631    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs podName:0ba46ac0-612c-4964-9454-8ae5e42bd728 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:34.373617421 +0000 UTC m=+2.723344814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs") pod "network-metrics-daemon-gbg4d" (UID: "0ba46ac0-612c-4964-9454-8ae5e42bd728") : secret "metrics-daemon-secret" not found
Oct 30 21:01:33.474008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:33.473971    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:33.474181 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:33.474083    4633 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:33.474181 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:33.474145    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:34.474131683 +0000 UTC m=+2.823859057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:34.380710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:34.380655    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:34.381080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:34.381060    4633 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:01:34.383187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:34.383164    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba46ac0-612c-4964-9454-8ae5e42bd728-metrics-certs\") pod \"network-metrics-daemon-gbg4d\" (UID: \"0ba46ac0-612c-4964-9454-8ae5e42bd728\") " pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:34.482084 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:34.482052    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:34.482218 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:34.482174    4633 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:34.482255 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:34.482235    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:36.482221941 +0000 UTC m=+4.831949319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:34.548743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:34.548713    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gbg4d"
Oct 30 21:01:34.816754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:34.816709    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gbg4d"]
Oct 30 21:01:34.820271 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:34.820230    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba46ac0_612c_4964_9454_8ae5e42bd728.slice/crio-2fddf2b94dc79df829cb0878efdd9e20e294216ca821e3b5c6a2485c5aaebd0b WatchSource:0}: Error finding container 2fddf2b94dc79df829cb0878efdd9e20e294216ca821e3b5c6a2485c5aaebd0b: Status 404 returned error can't find the container with id 2fddf2b94dc79df829cb0878efdd9e20e294216ca821e3b5c6a2485c5aaebd0b
Oct 30 21:01:34.822456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:34.822435    4633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:01:34.838775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:34.838751    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gbg4d" event={"ID":"0ba46ac0-612c-4964-9454-8ae5e42bd728","Type":"ContainerStarted","Data":"2fddf2b94dc79df829cb0878efdd9e20e294216ca821e3b5c6a2485c5aaebd0b"}
Oct 30 21:01:35.325247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:35.325207    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:36.181311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:36.181247    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:36.494804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:36.494721    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:36.494953 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:36.494845    4633 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:36.494953 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:36.494907    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls podName:941b3d3d-7780-4b2d-8ac9-950122fe66a5 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:40.494894147 +0000 UTC m=+8.844621520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b7hnm" (UID: "941b3d3d-7780-4b2d-8ac9-950122fe66a5") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:36.848462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:36.848377    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gbg4d" event={"ID":"0ba46ac0-612c-4964-9454-8ae5e42bd728","Type":"ContainerStarted","Data":"767815c854c839ba6157aa2ab910079197bd12a8e1ac13e693c9ecdde41e0ce3"}
Oct 30 21:01:36.848462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:36.848435    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gbg4d" event={"ID":"0ba46ac0-612c-4964-9454-8ae5e42bd728","Type":"ContainerStarted","Data":"28d7cc948314f55809c846c6e6658c898958fdcd12bf5b9c9bc3d13f235279f4"}
Oct 30 21:01:38.616631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:38.616592    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:01:40.519245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:40.519210    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:40.521755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:40.521730    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/941b3d3d-7780-4b2d-8ac9-950122fe66a5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7hnm\" (UID: \"941b3d3d-7780-4b2d-8ac9-950122fe66a5\") " pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:40.557631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:40.557605    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7hnm"
Oct 30 21:01:40.826868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:40.826794    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b7hnm"]
Oct 30 21:01:40.829972 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:40.829939    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941b3d3d_7780_4b2d_8ac9_950122fe66a5.slice/crio-d4ad007aedc5761b0d21eb3a1f506a4ad1aab8067143e3d9a797782f378814b4 WatchSource:0}: Error finding container d4ad007aedc5761b0d21eb3a1f506a4ad1aab8067143e3d9a797782f378814b4: Status 404 returned error can't find the container with id d4ad007aedc5761b0d21eb3a1f506a4ad1aab8067143e3d9a797782f378814b4
Oct 30 21:01:40.861265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:40.861237    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7hnm" event={"ID":"941b3d3d-7780-4b2d-8ac9-950122fe66a5","Type":"ContainerStarted","Data":"d4ad007aedc5761b0d21eb3a1f506a4ad1aab8067143e3d9a797782f378814b4"}
Oct 30 21:01:40.887292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:40.887269    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:40.887516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:40.887503    4633 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:41.327296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:41.327264    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5l9cm"
Oct 30 21:01:41.869049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:41.869010    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7hnm" event={"ID":"941b3d3d-7780-4b2d-8ac9-950122fe66a5","Type":"ContainerStarted","Data":"4c732af2297dcf98d63bf3779f5c041d942a87064bde351d15c41ac95a49c83f"}
Oct 30 21:01:41.869459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:41.869115    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7hnm" event={"ID":"941b3d3d-7780-4b2d-8ac9-950122fe66a5","Type":"ContainerStarted","Data":"d84b2bb59a8162e90b1eccaf58e0ca5c0cab69534902621545a26bc008f21e92"}
Oct 30 21:01:42.187496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:42.187466    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4cz8g"
Oct 30 21:01:43.877589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:43.877549    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7hnm" event={"ID":"941b3d3d-7780-4b2d-8ac9-950122fe66a5","Type":"ContainerStarted","Data":"aba1479e45112f55ec0543cc5dd3554c7f6968d3693a5c2f9e97cbc93eaba5e9"}
Oct 30 21:01:45.357243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.357210    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-bqkgp-68gbm"]
Oct 30 21:01:45.360620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.360595    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.377507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.377481    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"]
Oct 30 21:01:45.380912 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.380884    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7cf5k"]
Oct 30 21:01:45.381118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.380996    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"
Oct 30 21:01:45.383856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.383835    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bqkgp-68gbm"]
Oct 30 21:01:45.383996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.383976    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:01:45.392918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.392892    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:01:45.393030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.392935    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:01:45.393937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.393915    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 21:01:45.394968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.394086    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6hwrl\""
Oct 30 21:01:45.394968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.394507    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2svgj\""
Oct 30 21:01:45.395973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.395522    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"]
Oct 30 21:01:45.399056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.399035    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7cf5k"]
Oct 30 21:01:45.455716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.455664    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1d2ab1b7-4ef5-4d49-9aae-37aada030edb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rl755\" (UID: \"1d2ab1b7-4ef5-4d49-9aae-37aada030edb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"
Oct 30 21:01:45.455883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.455737    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-trusted-ca-bundle\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.455883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.455768    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvxz\" (UniqueName: \"kubernetes.io/projected/332e4903-5c8f-42c9-8de5-5a7fdfb81679-kube-api-access-6pvxz\") pod \"downloads-6dd789c8b7-7cf5k\" (UID: \"332e4903-5c8f-42c9-8de5-5a7fdfb81679\") " pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:01:45.455883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.455799    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4409113b-2c8a-4c21-b881-c0fc9cb12089-archives-path\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.456045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.455890    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-service-ca-bundle\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.456045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.455933    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5dl\" (UniqueName: \"kubernetes.io/projected/4409113b-2c8a-4c21-b881-c0fc9cb12089-kube-api-access-nt5dl\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.456045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.455960    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/332e4903-5c8f-42c9-8de5-5a7fdfb81679-tmp\") pod \"downloads-6dd789c8b7-7cf5k\" (UID: \"332e4903-5c8f-42c9-8de5-5a7fdfb81679\") " pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:01:45.458275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.458241    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-547b9cd8cc-j42s9"]
Oct 30 21:01:45.461243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.461228    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.469188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.469166    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:01:45.469391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.469276    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:01:45.469391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.469344    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:01:45.470766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.470745    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57d7c98f69-s5ln4"]
Oct 30 21:01:45.473638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.473615    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.473881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.473849    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-547b9cd8cc-j42s9"]
Oct 30 21:01:45.491757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.491722    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57d7c98f69-s5ln4"]
Oct 30 21:01:45.497696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.495650    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:01:45.497696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.496339    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:01:45.497696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.496406    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:01:45.497696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.496817    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8vks2\""
Oct 30 21:01:45.497696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.497022    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hl7k8\""
Oct 30 21:01:45.497696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.497332    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:01:45.497696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.497604    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:01:45.497696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.495649    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:01:45.557281 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557241    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-oauth-config\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.557281 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557277    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b029cd0b-f2be-438d-ac98-55a1e85eca2c-tmp\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.557511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557357    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-service-ca-bundle\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.557511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557389    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-config\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.557511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557414    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-image-registry-private-configuration\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.557511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557434    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5dl\" (UniqueName: \"kubernetes.io/projected/4409113b-2c8a-4c21-b881-c0fc9cb12089-kube-api-access-nt5dl\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.557511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557478    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-ca-trust-extracted\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.557798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557584    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-oauth-serving-cert\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.557798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557614    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-installation-pull-secrets\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.557798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557716    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvxz\" (UniqueName: \"kubernetes.io/projected/332e4903-5c8f-42c9-8de5-5a7fdfb81679-kube-api-access-6pvxz\") pod \"downloads-6dd789c8b7-7cf5k\" (UID: \"332e4903-5c8f-42c9-8de5-5a7fdfb81679\") " pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:01:45.557798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557742    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-trusted-ca\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.557798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557763    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4409113b-2c8a-4c21-b881-c0fc9cb12089-archives-path\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.557798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557783    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tx6v\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-kube-api-access-2tx6v\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.558092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557857    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/332e4903-5c8f-42c9-8de5-5a7fdfb81679-tmp\") pod \"downloads-6dd789c8b7-7cf5k\" (UID: \"332e4903-5c8f-42c9-8de5-5a7fdfb81679\") " pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:01:45.558092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557881    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/b029cd0b-f2be-438d-ac98-55a1e85eca2c-kube-api-access-rk8fp\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.558092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557905    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1d2ab1b7-4ef5-4d49-9aae-37aada030edb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rl755\" (UID: \"1d2ab1b7-4ef5-4d49-9aae-37aada030edb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"
Oct 30 21:01:45.558092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557930    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-trusted-ca-bundle\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.558092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.557962    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-bound-sa-token\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.558092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.558029    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-service-ca\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.558092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.558075    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-registry-tls\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.558401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.558104    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-registry-certificates\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.558401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.558146    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4409113b-2c8a-4c21-b881-c0fc9cb12089-archives-path\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.558401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.558160    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-serving-cert\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.558401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.558188    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/332e4903-5c8f-42c9-8de5-5a7fdfb81679-tmp\") pod \"downloads-6dd789c8b7-7cf5k\" (UID: \"332e4903-5c8f-42c9-8de5-5a7fdfb81679\") " pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:01:45.558401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.558290    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-service-ca-bundle\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.558801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.558779    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-trusted-ca-bundle\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.560519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.560499    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1d2ab1b7-4ef5-4d49-9aae-37aada030edb-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rl755\" (UID: \"1d2ab1b7-4ef5-4d49-9aae-37aada030edb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"
Oct 30 21:01:45.565701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.565658    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5dl\" (UniqueName: \"kubernetes.io/projected/4409113b-2c8a-4c21-b881-c0fc9cb12089-kube-api-access-nt5dl\") pod \"periodic-gathering-bqkgp-68gbm\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") " pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.565800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.565716    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvxz\" (UniqueName: \"kubernetes.io/projected/332e4903-5c8f-42c9-8de5-5a7fdfb81679-kube-api-access-6pvxz\") pod \"downloads-6dd789c8b7-7cf5k\" (UID: \"332e4903-5c8f-42c9-8de5-5a7fdfb81679\") " pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:01:45.659464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.659437    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-bound-sa-token\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.659648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.659473    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-service-ca\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.659648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.659489    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-registry-tls\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.659822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.659798    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-registry-certificates\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.659886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.659844    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-serving-cert\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.659886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.659880    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-oauth-config\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.659981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.659906    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b029cd0b-f2be-438d-ac98-55a1e85eca2c-tmp\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.660074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660043    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-config\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.660134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660108    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-image-registry-private-configuration\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.660193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660144    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-ca-trust-extracted\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.660193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660175    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-oauth-serving-cert\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.660298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660199    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-installation-pull-secrets\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.660298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660230    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-trusted-ca\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.660298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660235    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b029cd0b-f2be-438d-ac98-55a1e85eca2c-tmp\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.660298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660267    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tx6v\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-kube-api-access-2tx6v\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.660479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660322    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/b029cd0b-f2be-438d-ac98-55a1e85eca2c-kube-api-access-rk8fp\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.660479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660384    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-service-ca\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.660776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660755    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-ca-trust-extracted\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.660776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.660766    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-registry-certificates\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.661055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.661036    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-config\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.661125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.661055    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-oauth-serving-cert\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.661537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.661517    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-trusted-ca\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.663175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.663150    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-registry-tls\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.663280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.663200    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-oauth-config\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.663280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.663245    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-serving-cert\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.663389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.663341    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-image-registry-private-configuration\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.663586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.663571    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-installation-pull-secrets\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.667508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.667486    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-bound-sa-token\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.671975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.671949    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tx6v\" (UniqueName: \"kubernetes.io/projected/e1bcdfbe-ab13-4330-8e43-79b22175e1e5-kube-api-access-2tx6v\") pod \"image-registry-57d7c98f69-s5ln4\" (UID: \"e1bcdfbe-ab13-4330-8e43-79b22175e1e5\") " pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.672131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.672117    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/b029cd0b-f2be-438d-ac98-55a1e85eca2c-kube-api-access-rk8fp\") pod \"console-547b9cd8cc-j42s9\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") " pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.672756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.672740    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:01:45.694349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.694253    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"
Oct 30 21:01:45.701390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.701355    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:01:45.771940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.771909    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:45.785018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.784754    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:45.947452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.947382    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-bqkgp-68gbm"]
Oct 30 21:01:45.950311 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:45.950284    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4409113b_2c8a_4c21_b881_c0fc9cb12089.slice/crio-6b1a0954b3c3146305a4996cd75869430d8fd0911a0630cfe142fe362f93ec12 WatchSource:0}: Error finding container 6b1a0954b3c3146305a4996cd75869430d8fd0911a0630cfe142fe362f93ec12: Status 404 returned error can't find the container with id 6b1a0954b3c3146305a4996cd75869430d8fd0911a0630cfe142fe362f93ec12
Oct 30 21:01:45.976487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:45.976461    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"]
Oct 30 21:01:45.979587 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:45.979560    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2ab1b7_4ef5_4d49_9aae_37aada030edb.slice/crio-aa2da33cc6c1db2e1210499f0c4d6f939bd9d9d6ebddee41ec372741ef8eb911 WatchSource:0}: Error finding container aa2da33cc6c1db2e1210499f0c4d6f939bd9d9d6ebddee41ec372741ef8eb911: Status 404 returned error can't find the container with id aa2da33cc6c1db2e1210499f0c4d6f939bd9d9d6ebddee41ec372741ef8eb911
Oct 30 21:01:46.005313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.005288    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7cf5k"]
Oct 30 21:01:46.009157 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:46.009134    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332e4903_5c8f_42c9_8de5_5a7fdfb81679.slice/crio-81b6b70e4382d0eb81a9d5f5314ef9725f7b8d633625554cb9b1a06910600dad WatchSource:0}: Error finding container 81b6b70e4382d0eb81a9d5f5314ef9725f7b8d633625554cb9b1a06910600dad: Status 404 returned error can't find the container with id 81b6b70e4382d0eb81a9d5f5314ef9725f7b8d633625554cb9b1a06910600dad
Oct 30 21:01:46.055581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.055490    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-547b9cd8cc-j42s9"]
Oct 30 21:01:46.061560 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:46.061521    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb029cd0b_f2be_438d_ac98_55a1e85eca2c.slice/crio-4262558df30184df9162dea1196b829d8f7edb4a70cc37e179ca61042a4030a8 WatchSource:0}: Error finding container 4262558df30184df9162dea1196b829d8f7edb4a70cc37e179ca61042a4030a8: Status 404 returned error can't find the container with id 4262558df30184df9162dea1196b829d8f7edb4a70cc37e179ca61042a4030a8
Oct 30 21:01:46.078420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.076108    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57d7c98f69-s5ln4"]
Oct 30 21:01:46.081232 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:46.081201    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bcdfbe_ab13_4330_8e43_79b22175e1e5.slice/crio-b6c2dc257ae7a2aecfebf4bffd050138d72a13810e8895eb6b6c34ce405ad5de WatchSource:0}: Error finding container b6c2dc257ae7a2aecfebf4bffd050138d72a13810e8895eb6b6c34ce405ad5de: Status 404 returned error can't find the container with id b6c2dc257ae7a2aecfebf4bffd050138d72a13810e8895eb6b6c34ce405ad5de
Oct 30 21:01:46.778546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.778504    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5l9cm_243d8065-2175-4a20-b229-ebaae8e987fb/dns/0.log"
Oct 30 21:01:46.895191 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.895146    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547b9cd8cc-j42s9" event={"ID":"b029cd0b-f2be-438d-ac98-55a1e85eca2c","Type":"ContainerStarted","Data":"4262558df30184df9162dea1196b829d8f7edb4a70cc37e179ca61042a4030a8"}
Oct 30 21:01:46.898211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.898178    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4" event={"ID":"e1bcdfbe-ab13-4330-8e43-79b22175e1e5","Type":"ContainerStarted","Data":"98fd7a5bf5a9ec8e07fa4d62e9fce2abcccc9ea203d021aeb42ecab57130ed9d"}
Oct 30 21:01:46.898357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.898233    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4" event={"ID":"e1bcdfbe-ab13-4330-8e43-79b22175e1e5","Type":"ContainerStarted","Data":"b6c2dc257ae7a2aecfebf4bffd050138d72a13810e8895eb6b6c34ce405ad5de"}
Oct 30 21:01:46.898986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.898966    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:01:46.900915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.900851    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7cf5k" event={"ID":"332e4903-5c8f-42c9-8de5-5a7fdfb81679","Type":"ContainerStarted","Data":"81b6b70e4382d0eb81a9d5f5314ef9725f7b8d633625554cb9b1a06910600dad"}
Oct 30 21:01:46.903065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.903037    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bqkgp-68gbm" event={"ID":"4409113b-2c8a-4c21-b881-c0fc9cb12089","Type":"ContainerStarted","Data":"a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f"}
Oct 30 21:01:46.903704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.903683    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bqkgp-68gbm" event={"ID":"4409113b-2c8a-4c21-b881-c0fc9cb12089","Type":"ContainerStarted","Data":"6b1a0954b3c3146305a4996cd75869430d8fd0911a0630cfe142fe362f93ec12"}
Oct 30 21:01:46.905254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.905223    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755" event={"ID":"1d2ab1b7-4ef5-4d49-9aae-37aada030edb","Type":"ContainerStarted","Data":"aa2da33cc6c1db2e1210499f0c4d6f939bd9d9d6ebddee41ec372741ef8eb911"}
Oct 30 21:01:46.919837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.919776    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4" podStartSLOduration=1.9197323530000001 podStartE2EDuration="1.919732353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:46.918081011 +0000 UTC m=+15.267808408" watchObservedRunningTime="2026-10-30 21:01:46.919732353 +0000 UTC m=+15.269459748"
Oct 30 21:01:46.964021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:46.963916    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5l9cm_243d8065-2175-4a20-b229-ebaae8e987fb/kube-rbac-proxy/0.log"
Oct 30 21:01:47.912124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:47.912086    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755" event={"ID":"1d2ab1b7-4ef5-4d49-9aae-37aada030edb","Type":"ContainerStarted","Data":"03f0697b51a2defecb51c0cf6f3fbeff2f48bd3d5fdf2f2f85bf3fb0b8e0a8bc"}
Oct 30 21:01:47.927611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:47.927558    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-bqkgp-68gbm" podStartSLOduration=2.9275391649999998 podStartE2EDuration="2.927539165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:46.935960552 +0000 UTC m=+15.285687947" watchObservedRunningTime="2026-10-30 21:01:47.927539165 +0000 UTC m=+16.277266561"
Oct 30 21:01:47.928006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:47.927982    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755" podStartSLOduration=1.550101274 podStartE2EDuration="2.927973639s" totalImagesPullingTime="1.377872365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:47.926936486 +0000 UTC m=+16.276663883" watchObservedRunningTime="2026-10-30 21:01:47.927973639 +0000 UTC m=+16.277701032"
Oct 30 21:01:47.964415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:47.964383    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r7gpg_2492dc3b-9893-45f5-af15-314312a0a18f/dns-node-resolver/0.log"
Oct 30 21:01:48.162828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.162645    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57d7c98f69-s5ln4_e1bcdfbe-ab13-4330-8e43-79b22175e1e5/registry/0.log"
Oct 30 21:01:48.171389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.171344    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b755fddbb-qd8t4"]
Oct 30 21:01:48.174992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.174968    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.183569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.183541    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:01:48.184597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.184568    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b755fddbb-qd8t4"]
Oct 30 21:01:48.282608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.282573    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-serving-cert\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.282786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.282622    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-config\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.282786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.282663    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-oauth-serving-cert\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.282786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.282772    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-oauth-config\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.282987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.282790    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c77896-1ab6-4b63-9b09-377b9b7997de-tmp\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.282987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.282812    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-trusted-ca-bundle\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.282987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.282901    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-service-ca\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.282987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.282952    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74db\" (UniqueName: \"kubernetes.io/projected/d5c77896-1ab6-4b63-9b09-377b9b7997de-kube-api-access-j74db\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.383576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.383534    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-service-ca\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.383784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.383588    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74db\" (UniqueName: \"kubernetes.io/projected/d5c77896-1ab6-4b63-9b09-377b9b7997de-kube-api-access-j74db\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.383784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.383633    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-serving-cert\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.383784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.383684    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-config\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.383784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.383709    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-oauth-serving-cert\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.383784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.383762    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-oauth-config\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.384033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.383786    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c77896-1ab6-4b63-9b09-377b9b7997de-tmp\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.384033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.383843    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-trusted-ca-bundle\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.384912 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.384885    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-trusted-ca-bundle\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.385522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.385499    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-service-ca\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.390128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.390095    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c77896-1ab6-4b63-9b09-377b9b7997de-tmp\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.390708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.390688    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-oauth-serving-cert\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.391697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.391284    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-config\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.393343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.393316    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-serving-cert\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.395926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.395902    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-oauth-config\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.399071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.399047    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74db\" (UniqueName: \"kubernetes.io/projected/d5c77896-1ab6-4b63-9b09-377b9b7997de-kube-api-access-j74db\") pod \"console-6b755fddbb-qd8t4\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") " pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.491908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.491799    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:48.786928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.786848    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b755fddbb-qd8t4"]
Oct 30 21:01:48.790330 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:48.790299    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c77896_1ab6_4b63_9b09_377b9b7997de.slice/crio-72bce9ce0fd6188330b2a5cf95069dbd9f39df37f37735ddc5902877d36a837f WatchSource:0}: Error finding container 72bce9ce0fd6188330b2a5cf95069dbd9f39df37f37735ddc5902877d36a837f: Status 404 returned error can't find the container with id 72bce9ce0fd6188330b2a5cf95069dbd9f39df37f37735ddc5902877d36a837f
Oct 30 21:01:48.917740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.917649    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b755fddbb-qd8t4" event={"ID":"d5c77896-1ab6-4b63-9b09-377b9b7997de","Type":"ContainerStarted","Data":"72bce9ce0fd6188330b2a5cf95069dbd9f39df37f37735ddc5902877d36a837f"}
Oct 30 21:01:48.918455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.918423    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"
Oct 30 21:01:48.924108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:48.923901    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rl755"
Oct 30 21:01:49.160949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.159616    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"]
Oct 30 21:01:49.168689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.168634    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.169659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.169640    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wrffd_595d39f4-1364-4e9e-87ae-09120d783bdd/node-ca/0.log"
Oct 30 21:01:49.174479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.174447    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:01:49.174829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.174774    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:01:49.175439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.175419    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:01:49.188377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.188360    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:01:49.193152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.192251    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"]
Oct 30 21:01:49.284395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.284362    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:01:49.284577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.284411    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bqwjl\""
Oct 30 21:01:49.291316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.291284    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.291504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.291349    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.291504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.291385    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003c3612-162c-445f-a595-2c1bcb37f9b2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.291504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.291417    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbdh\" (UniqueName: \"kubernetes.io/projected/003c3612-162c-445f-a595-2c1bcb37f9b2-kube-api-access-hrbdh\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.391869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.391830    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.392054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.391933    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.392054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.391964    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003c3612-162c-445f-a595-2c1bcb37f9b2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.392054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.391986    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbdh\" (UniqueName: \"kubernetes.io/projected/003c3612-162c-445f-a595-2c1bcb37f9b2-kube-api-access-hrbdh\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.392054 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:49.392006    4633 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Oct 30 21:01:49.392244 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:49.392088    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-tls podName:003c3612-162c-445f-a595-2c1bcb37f9b2 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:49.89206499 +0000 UTC m=+18.241792378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-tls") pod "prometheus-operator-9cd89b74c-7pp58" (UID: "003c3612-162c-445f-a595-2c1bcb37f9b2") : secret "prometheus-operator-tls" not found
Oct 30 21:01:49.392913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.392888    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/003c3612-162c-445f-a595-2c1bcb37f9b2-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.395027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.395006    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.401161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.401139    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbdh\" (UniqueName: \"kubernetes.io/projected/003c3612-162c-445f-a595-2c1bcb37f9b2-kube-api-access-hrbdh\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.563450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.563419    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jq4p6_205f3154-c424-4dff-9536-cff842b7eac1/serve-healthcheck-canary/0.log"
Oct 30 21:01:49.896559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.896473    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.908997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.908967    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/003c3612-162c-445f-a595-2c1bcb37f9b2-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-7pp58\" (UID: \"003c3612-162c-445f-a595-2c1bcb37f9b2\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:49.925448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:49.925411    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547b9cd8cc-j42s9" event={"ID":"b029cd0b-f2be-438d-ac98-55a1e85eca2c","Type":"ContainerStarted","Data":"73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4"}
Oct 30 21:01:50.086591 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:50.086325    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"
Oct 30 21:01:50.367994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:50.367925    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-547b9cd8cc-j42s9" podStartSLOduration=1.992039887 podStartE2EDuration="5.367904869s" totalImagesPullingTime="3.375864982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:49.946057555 +0000 UTC m=+18.295784952" watchObservedRunningTime="2026-10-30 21:01:50.367904869 +0000 UTC m=+18.717632265"
Oct 30 21:01:50.368337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:50.368305    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-7pp58"]
Oct 30 21:01:50.372310 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:50.372283    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003c3612_162c_445f_a595_2c1bcb37f9b2.slice/crio-b147efb24a7f1d1599931aa1a02a78a464a1d8687d302a88ac661f14ebe4680f WatchSource:0}: Error finding container b147efb24a7f1d1599931aa1a02a78a464a1d8687d302a88ac661f14ebe4680f: Status 404 returned error can't find the container with id b147efb24a7f1d1599931aa1a02a78a464a1d8687d302a88ac661f14ebe4680f
Oct 30 21:01:50.931529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:50.930966    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b755fddbb-qd8t4" event={"ID":"d5c77896-1ab6-4b63-9b09-377b9b7997de","Type":"ContainerStarted","Data":"bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940"}
Oct 30 21:01:50.932431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:50.932393    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58" event={"ID":"003c3612-162c-445f-a595-2c1bcb37f9b2","Type":"ContainerStarted","Data":"b147efb24a7f1d1599931aa1a02a78a464a1d8687d302a88ac661f14ebe4680f"}
Oct 30 21:01:50.948329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:50.948273    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b755fddbb-qd8t4" podStartSLOduration=1.06653949 podStartE2EDuration="2.948257993s" totalImagesPullingTime="1.881718503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:50.94808402 +0000 UTC m=+19.297811419" watchObservedRunningTime="2026-10-30 21:01:50.948257993 +0000 UTC m=+19.297985390"
Oct 30 21:01:51.939813 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:51.939779    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58" event={"ID":"003c3612-162c-445f-a595-2c1bcb37f9b2","Type":"ContainerStarted","Data":"f0f47ce1284178c7d5032f866ae4dbf62879b84f776eedd2fa91c1e25e726d6a"}
Oct 30 21:01:52.946490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:52.946448    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58" event={"ID":"003c3612-162c-445f-a595-2c1bcb37f9b2","Type":"ContainerStarted","Data":"3dd94240bf974f39c85d3de67b882637d29cc20ad096b65e1dc33e5761f8b0b3"}
Oct 30 21:01:52.968130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:52.968071    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-7pp58" podStartSLOduration=2.560915574 podStartE2EDuration="3.968052848s" totalImagesPullingTime="1.407137274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:52.966097394 +0000 UTC m=+21.315824791" watchObservedRunningTime="2026-10-30 21:01:52.968052848 +0000 UTC m=+21.317780244"
Oct 30 21:01:54.553978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.553936    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9vfdj"]
Oct 30 21:01:54.559185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.559160    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.577486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.577459    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:01:54.577634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.577550    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:01:54.577809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.577737    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4f5fg\""
Oct 30 21:01:54.637986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.637934    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:01:54.646204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646157    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97ac0ce7-c6e3-445a-9492-4976098b86f9-metrics-client-ca\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.646414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646273    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-sys\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.646414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646366    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kql\" (UniqueName: \"kubernetes.io/projected/97ac0ce7-c6e3-445a-9492-4976098b86f9-kube-api-access-64kql\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.646565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646425    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.646565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646464    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-root\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.646565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646555    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-wtmp\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.646715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646594    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-textfile\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.646715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646614    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-tls\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.646791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.646723    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748093    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-textfile\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748142    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-tls\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748270    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748326    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97ac0ce7-c6e3-445a-9492-4976098b86f9-metrics-client-ca\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748388    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-sys\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748413    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kql\" (UniqueName: \"kubernetes.io/projected/97ac0ce7-c6e3-445a-9492-4976098b86f9-kube-api-access-64kql\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748451    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748485    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-root\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748493    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-sys\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748510    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-wtmp\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748521    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-textfile\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748658    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-wtmp\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.748894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.748803    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97ac0ce7-c6e3-445a-9492-4976098b86f9-root\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.749226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.749204    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97ac0ce7-c6e3-445a-9492-4976098b86f9-metrics-client-ca\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.749707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.749654    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.751432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.751412    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.751546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.751439    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97ac0ce7-c6e3-445a-9492-4976098b86f9-node-exporter-tls\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.785663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.785586    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kql\" (UniqueName: \"kubernetes.io/projected/97ac0ce7-c6e3-445a-9492-4976098b86f9-kube-api-access-64kql\") pod \"node-exporter-9vfdj\" (UID: \"97ac0ce7-c6e3-445a-9492-4976098b86f9\") " pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.879562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.879476    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9vfdj"
Oct 30 21:01:54.923880 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:01:54.923851    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ac0ce7_c6e3_445a_9492_4976098b86f9.slice/crio-971adcebd5e727bb2e5ee2c79a76033cb44f21dfec5d3ecc1aadbc76f289c870 WatchSource:0}: Error finding container 971adcebd5e727bb2e5ee2c79a76033cb44f21dfec5d3ecc1aadbc76f289c870: Status 404 returned error can't find the container with id 971adcebd5e727bb2e5ee2c79a76033cb44f21dfec5d3ecc1aadbc76f289c870
Oct 30 21:01:54.955513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:54.955480    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vfdj" event={"ID":"97ac0ce7-c6e3-445a-9492-4976098b86f9","Type":"ContainerStarted","Data":"971adcebd5e727bb2e5ee2c79a76033cb44f21dfec5d3ecc1aadbc76f289c870"}
Oct 30 21:01:55.021977 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:01:55.021935    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:01:55.772724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:55.772609    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:55.773300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:55.772697    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:55.779242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:55.779212    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:55.965393 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:55.965360    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:01:56.965765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:56.965725    4633 generic.go:356] "Generic (PLEG): container finished" podID="97ac0ce7-c6e3-445a-9492-4976098b86f9" containerID="6584a299eb6ed01b11baa6ff6f35aa406f82ca1c2437a2ec7e6795008008f8bc" exitCode=0
Oct 30 21:01:56.966275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:56.965819    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vfdj" event={"ID":"97ac0ce7-c6e3-445a-9492-4976098b86f9","Type":"ContainerDied","Data":"6584a299eb6ed01b11baa6ff6f35aa406f82ca1c2437a2ec7e6795008008f8bc"}
Oct 30 21:01:57.972569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:57.972526    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vfdj" event={"ID":"97ac0ce7-c6e3-445a-9492-4976098b86f9","Type":"ContainerStarted","Data":"20c298be97afc2f727f46e6d694a9c6b2c0796da47961ab9ced587bcecf5093c"}
Oct 30 21:01:57.973261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:57.972594    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vfdj" event={"ID":"97ac0ce7-c6e3-445a-9492-4976098b86f9","Type":"ContainerStarted","Data":"bf3fbd6dd619f1d2a3b900f4e6cb410b80a7b6a5b47de4f7b3bc4a6f95f39aad"}
Oct 30 21:01:57.992893 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:57.992843    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9vfdj" podStartSLOduration=2.384691885 podStartE2EDuration="3.992829855s" totalImagesPullingTime="1.511285655s" totalInitContainerRuntime="96.852315ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:57.991850724 +0000 UTC m=+26.341578119" watchObservedRunningTime="2026-10-30 21:01:57.992829855 +0000 UTC m=+26.342557248"
Oct 30 21:01:58.492357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:58.492317    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:58.492556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:58.492393    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:58.497728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:58.497691    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:58.981031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:58.980999    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:01:59.034450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:01:59.034413    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-547b9cd8cc-j42s9"]
Oct 30 21:02:02.548410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:02.548382    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r2kvg"
Oct 30 21:02:03.998903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:03.998856    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7cf5k" event={"ID":"332e4903-5c8f-42c9-8de5-5a7fdfb81679","Type":"ContainerStarted","Data":"5eb8f1a9b03e3531efda7a60301b6846a3b4c647df75565977a738ef604c9825"}
Oct 30 21:02:03.999390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:03.999103    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:02:04.014124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.014088    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7cf5k"
Oct 30 21:02:04.018630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.018574    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-7cf5k" podStartSLOduration=1.649074183 podStartE2EDuration="19.018557s" totalImagesPullingTime="17.369482817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:04.017084616 +0000 UTC m=+32.366812012" watchObservedRunningTime="2026-10-30 21:02:04.018557 +0000 UTC m=+32.368284397"
Oct 30 21:02:04.381707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.381607    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c796f96cd-mcnl7"]
Oct 30 21:02:04.418149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.418112    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c796f96cd-mcnl7"]
Oct 30 21:02:04.418327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.418243    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.548459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.548415    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-trusted-ca-bundle\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.548653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.548486    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-serving-cert\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.548653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.548599    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-service-ca\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.548799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.548659    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-config\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.548799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.548710    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-oauth-serving-cert\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.548904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.548835    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2zt\" (UniqueName: \"kubernetes.io/projected/835ad90d-1857-4ec2-80ad-6c20e612ba11-kube-api-access-jl2zt\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.548904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.548873    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/835ad90d-1857-4ec2-80ad-6c20e612ba11-tmp\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.548904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.548895    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-oauth-config\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.649511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.649426    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-service-ca\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.649511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.649487    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-config\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.649801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.649515    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-oauth-serving-cert\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.649801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.649562    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2zt\" (UniqueName: \"kubernetes.io/projected/835ad90d-1857-4ec2-80ad-6c20e612ba11-kube-api-access-jl2zt\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.649801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.649590    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/835ad90d-1857-4ec2-80ad-6c20e612ba11-tmp\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.649801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.649620    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-oauth-config\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.649801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.649663    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-trusted-ca-bundle\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.649801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.649726    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-serving-cert\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.650353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.650320    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-config\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.650461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.650359    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-oauth-serving-cert\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.650518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.650467    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-service-ca\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.650797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.650765    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/835ad90d-1857-4ec2-80ad-6c20e612ba11-tmp\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.651421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.651390    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-trusted-ca-bundle\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.652821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.652798    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-serving-cert\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.653233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.653212    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-oauth-config\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.658915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.658893    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2zt\" (UniqueName: \"kubernetes.io/projected/835ad90d-1857-4ec2-80ad-6c20e612ba11-kube-api-access-jl2zt\") pod \"console-7c796f96cd-mcnl7\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") " pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:04.746340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:04.746298    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:05.047486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:05.047407    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c796f96cd-mcnl7"]
Oct 30 21:02:05.050768 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:02:05.050716    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835ad90d_1857_4ec2_80ad_6c20e612ba11.slice/crio-a976a4719f22d02d3c500563b596624d92dd918b756e076aa674bf0a335ed0b0 WatchSource:0}: Error finding container a976a4719f22d02d3c500563b596624d92dd918b756e076aa674bf0a335ed0b0: Status 404 returned error can't find the container with id a976a4719f22d02d3c500563b596624d92dd918b756e076aa674bf0a335ed0b0
Oct 30 21:02:06.010749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:06.010707    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c796f96cd-mcnl7" event={"ID":"835ad90d-1857-4ec2-80ad-6c20e612ba11","Type":"ContainerStarted","Data":"bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f"}
Oct 30 21:02:06.010948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:06.010770    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c796f96cd-mcnl7" event={"ID":"835ad90d-1857-4ec2-80ad-6c20e612ba11","Type":"ContainerStarted","Data":"a976a4719f22d02d3c500563b596624d92dd918b756e076aa674bf0a335ed0b0"}
Oct 30 21:02:06.030198 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:06.030138    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c796f96cd-mcnl7" podStartSLOduration=2.030120421 podStartE2EDuration="2.030120421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:06.028199612 +0000 UTC m=+34.377927011" watchObservedRunningTime="2026-10-30 21:02:06.030120421 +0000 UTC m=+34.379847892"
Oct 30 21:02:08.620581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:08.620548    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kwb4j"
Oct 30 21:02:08.934092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:08.934061    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57d7c98f69-s5ln4"
Oct 30 21:02:14.747029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:14.746942    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:14.747029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:14.747025    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:14.751996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:14.751975    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:15.063689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:15.063584    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:02:15.115490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:15.115451    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b755fddbb-qd8t4"]
Oct 30 21:02:17.605341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.605305    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rtfgg-7ntt6"]
Oct 30 21:02:17.605783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.605542    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6" podUID="1c66adeb-03be-476e-960e-2544b9605375" containerName="insights-gathering" containerID="cri-o://ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079" gracePeriod=30
Oct 30 21:02:17.880890 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.880862    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:02:17.975448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.975415    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path\") pod \"1c66adeb-03be-476e-960e-2544b9605375\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") "
Oct 30 21:02:17.975448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.975455    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle\") pod \"1c66adeb-03be-476e-960e-2544b9605375\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") "
Oct 30 21:02:17.975721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.975491    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6\" (UniqueName: \"kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6\") pod \"1c66adeb-03be-476e-960e-2544b9605375\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") "
Oct 30 21:02:17.975721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.975519    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle\") pod \"1c66adeb-03be-476e-960e-2544b9605375\" (UID: \"1c66adeb-03be-476e-960e-2544b9605375\") "
Oct 30 21:02:17.975865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.975843    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path" pod "1c66adeb-03be-476e-960e-2544b9605375" (UID: "1c66adeb-03be-476e-960e-2544b9605375"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:17.975933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.975852    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle" pod "1c66adeb-03be-476e-960e-2544b9605375" (UID: "1c66adeb-03be-476e-960e-2544b9605375"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:17.976057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.976035    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle" pod "1c66adeb-03be-476e-960e-2544b9605375" (UID: "1c66adeb-03be-476e-960e-2544b9605375"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:17.978001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:17.977980    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6" pod "1c66adeb-03be-476e-960e-2544b9605375" (UID: "1c66adeb-03be-476e-960e-2544b9605375"). InnerVolumeSpecName "kube-api-access-7xzx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:18.070782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.070746    4633 generic.go:356] "Generic (PLEG): container finished" podID="1c66adeb-03be-476e-960e-2544b9605375" containerID="ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079" exitCode=2
Oct 30 21:02:18.070957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.070807    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6"
Oct 30 21:02:18.070957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.070831    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6" event={"ID":"1c66adeb-03be-476e-960e-2544b9605375","Type":"ContainerDied","Data":"ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079"}
Oct 30 21:02:18.070957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.070887    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rtfgg-7ntt6" event={"ID":"1c66adeb-03be-476e-960e-2544b9605375","Type":"ContainerDied","Data":"98cb2cc90ab939e3a55004b4d377ecabba05d2f7203fc8b4ca4f4d335f7826d3"}
Oct 30 21:02:18.070957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.070907    4633 scope.go:122] "RemoveContainer" containerID="ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079"
Oct 30 21:02:18.076176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.076152    4633 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1c66adeb-03be-476e-960e-2544b9605375-archives-path\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.076271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.076181    4633 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-service-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.076271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.076199    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzx6\" (UniqueName: \"kubernetes.io/projected/1c66adeb-03be-476e-960e-2544b9605375-kube-api-access-7xzx6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.076271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.076215    4633 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c66adeb-03be-476e-960e-2544b9605375-trusted-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:18.080047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.080030    4633 scope.go:122] "RemoveContainer" containerID="ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079"
Oct 30 21:02:18.080354 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:02:18.080332    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079\": container with ID starting with ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079 not found: ID does not exist" containerID="ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079"
Oct 30 21:02:18.080414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.080369    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079"} err="failed to get container status \"ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079\": rpc error: code = NotFound desc = could not find container \"ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079\": container with ID starting with ec7344f3fa14d623e8d7ffb35d6249f3f8969b305b489dca41e8e2fd778bf079 not found: ID does not exist"
Oct 30 21:02:18.091181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.091157    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rtfgg-7ntt6"]
Oct 30 21:02:18.095543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:18.095524    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-rtfgg-7ntt6"]
Oct 30 21:02:19.787551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:19.787518    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c66adeb-03be-476e-960e-2544b9605375" path="/var/lib/kubelet/pods/1c66adeb-03be-476e-960e-2544b9605375/volumes"
Oct 30 21:02:24.057633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.057568    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-547b9cd8cc-j42s9" podUID="b029cd0b-f2be-438d-ac98-55a1e85eca2c" containerName="console" containerID="cri-o://73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4" gracePeriod=15
Oct 30 21:02:24.337722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.337693    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:02:24.427136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427100    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b029cd0b-f2be-438d-ac98-55a1e85eca2c-kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/b029cd0b-f2be-438d-ac98-55a1e85eca2c-kube-api-access-rk8fp\") pod \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") "
Oct 30 21:02:24.427136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427142    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-service-ca\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-service-ca\") pod \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") "
Oct 30 21:02:24.427381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427211    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b029cd0b-f2be-438d-ac98-55a1e85eca2c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b029cd0b-f2be-438d-ac98-55a1e85eca2c-tmp\") pod \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") "
Oct 30 21:02:24.427381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427237    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-config\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-config\") pod \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") "
Oct 30 21:02:24.427381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427271    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-oauth-serving-cert\") pod \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") "
Oct 30 21:02:24.427381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427313    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-oauth-config\") pod \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") "
Oct 30 21:02:24.427381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427342    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-serving-cert\") pod \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\" (UID: \"b029cd0b-f2be-438d-ac98-55a1e85eca2c\") "
Oct 30 21:02:24.427618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427555    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b029cd0b-f2be-438d-ac98-55a1e85eca2c-tmp" pod "b029cd0b-f2be-438d-ac98-55a1e85eca2c" (UID: "b029cd0b-f2be-438d-ac98-55a1e85eca2c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:24.427703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427642    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-config" pod "b029cd0b-f2be-438d-ac98-55a1e85eca2c" (UID: "b029cd0b-f2be-438d-ac98-55a1e85eca2c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:24.427770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427718    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-service-ca" pod "b029cd0b-f2be-438d-ac98-55a1e85eca2c" (UID: "b029cd0b-f2be-438d-ac98-55a1e85eca2c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:24.427841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.427764    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-oauth-serving-cert" pod "b029cd0b-f2be-438d-ac98-55a1e85eca2c" (UID: "b029cd0b-f2be-438d-ac98-55a1e85eca2c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:24.429651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.429623    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-serving-cert" pod "b029cd0b-f2be-438d-ac98-55a1e85eca2c" (UID: "b029cd0b-f2be-438d-ac98-55a1e85eca2c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:24.429651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.429644    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-oauth-config" pod "b029cd0b-f2be-438d-ac98-55a1e85eca2c" (UID: "b029cd0b-f2be-438d-ac98-55a1e85eca2c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:24.429867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.429700    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b029cd0b-f2be-438d-ac98-55a1e85eca2c-kube-api-access-rk8fp" pod "b029cd0b-f2be-438d-ac98-55a1e85eca2c" (UID: "b029cd0b-f2be-438d-ac98-55a1e85eca2c"). InnerVolumeSpecName "kube-api-access-rk8fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:24.528869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.528797    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/b029cd0b-f2be-438d-ac98-55a1e85eca2c-kube-api-access-rk8fp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:24.528869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.528827    4633 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:24.528869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.528838    4633 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b029cd0b-f2be-438d-ac98-55a1e85eca2c-tmp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:24.528869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.528846    4633 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-config\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:24.528869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.528856    4633 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b029cd0b-f2be-438d-ac98-55a1e85eca2c-oauth-serving-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:24.528869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.528865    4633 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-oauth-config\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:24.528869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:24.528874    4633 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b029cd0b-f2be-438d-ac98-55a1e85eca2c-console-serving-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:25.013044 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:02:25.013009    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:02:25.099239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.099205    4633 generic.go:356] "Generic (PLEG): container finished" podID="b029cd0b-f2be-438d-ac98-55a1e85eca2c" containerID="73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4" exitCode=0
Oct 30 21:02:25.099700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.099257    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547b9cd8cc-j42s9"
Oct 30 21:02:25.099700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.099298    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547b9cd8cc-j42s9" event={"ID":"b029cd0b-f2be-438d-ac98-55a1e85eca2c","Type":"ContainerDied","Data":"73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4"}
Oct 30 21:02:25.099700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.099359    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547b9cd8cc-j42s9" event={"ID":"b029cd0b-f2be-438d-ac98-55a1e85eca2c","Type":"ContainerDied","Data":"4262558df30184df9162dea1196b829d8f7edb4a70cc37e179ca61042a4030a8"}
Oct 30 21:02:25.099700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.099380    4633 scope.go:122] "RemoveContainer" containerID="73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4"
Oct 30 21:02:25.108472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.108451    4633 scope.go:122] "RemoveContainer" containerID="73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4"
Oct 30 21:02:25.108795 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:02:25.108776    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4\": container with ID starting with 73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4 not found: ID does not exist" containerID="73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4"
Oct 30 21:02:25.108876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.108808    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4"} err="failed to get container status \"73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4\": rpc error: code = NotFound desc = could not find container \"73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4\": container with ID starting with 73a1fe7efb7aabf4595dd44e1e7dcee89db7ac7f1485b703d413054da1f0dba4 not found: ID does not exist"
Oct 30 21:02:25.120986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.120960    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-547b9cd8cc-j42s9"]
Oct 30 21:02:25.122985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.122966    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-547b9cd8cc-j42s9"]
Oct 30 21:02:25.787716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:25.787684    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b029cd0b-f2be-438d-ac98-55a1e85eca2c" path="/var/lib/kubelet/pods/b029cd0b-f2be-438d-ac98-55a1e85eca2c/volumes"
Oct 30 21:02:40.137051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.136984    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6b755fddbb-qd8t4" podUID="d5c77896-1ab6-4b63-9b09-377b9b7997de" containerName="console" containerID="cri-o://bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940" gracePeriod=15
Oct 30 21:02:40.419422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.419394    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:02:40.566553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.566516    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-oauth-config\") pod \"d5c77896-1ab6-4b63-9b09-377b9b7997de\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") "
Oct 30 21:02:40.566553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.566560    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-trusted-ca-bundle\") pod \"d5c77896-1ab6-4b63-9b09-377b9b7997de\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") "
Oct 30 21:02:40.566848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.566592    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-service-ca\") pod \"d5c77896-1ab6-4b63-9b09-377b9b7997de\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") "
Oct 30 21:02:40.566848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.566623    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-config\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-config\") pod \"d5c77896-1ab6-4b63-9b09-377b9b7997de\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") "
Oct 30 21:02:40.566848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.566785    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-serving-cert\") pod \"d5c77896-1ab6-4b63-9b09-377b9b7997de\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") "
Oct 30 21:02:40.567006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.566876    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c77896-1ab6-4b63-9b09-377b9b7997de-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c77896-1ab6-4b63-9b09-377b9b7997de-tmp\") pod \"d5c77896-1ab6-4b63-9b09-377b9b7997de\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") "
Oct 30 21:02:40.567006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.566918    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-oauth-serving-cert\") pod \"d5c77896-1ab6-4b63-9b09-377b9b7997de\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") "
Oct 30 21:02:40.567006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.566973    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c77896-1ab6-4b63-9b09-377b9b7997de-kube-api-access-j74db\" (UniqueName: \"kubernetes.io/projected/d5c77896-1ab6-4b63-9b09-377b9b7997de-kube-api-access-j74db\") pod \"d5c77896-1ab6-4b63-9b09-377b9b7997de\" (UID: \"d5c77896-1ab6-4b63-9b09-377b9b7997de\") "
Oct 30 21:02:40.567143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567030    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-trusted-ca-bundle" pod "d5c77896-1ab6-4b63-9b09-377b9b7997de" (UID: "d5c77896-1ab6-4b63-9b09-377b9b7997de"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:40.567143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567063    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-service-ca" pod "d5c77896-1ab6-4b63-9b09-377b9b7997de" (UID: "d5c77896-1ab6-4b63-9b09-377b9b7997de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:40.567143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567078    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-config" pod "d5c77896-1ab6-4b63-9b09-377b9b7997de" (UID: "d5c77896-1ab6-4b63-9b09-377b9b7997de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:40.567338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567195    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c77896-1ab6-4b63-9b09-377b9b7997de-tmp" pod "d5c77896-1ab6-4b63-9b09-377b9b7997de" (UID: "d5c77896-1ab6-4b63-9b09-377b9b7997de"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:40.567402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567354    4633 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c77896-1ab6-4b63-9b09-377b9b7997de-tmp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.567402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567377    4633 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-trusted-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.567402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567386    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-oauth-serving-cert" pod "d5c77896-1ab6-4b63-9b09-377b9b7997de" (UID: "d5c77896-1ab6-4b63-9b09-377b9b7997de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:40.567402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567395    4633 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.567548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.567430    4633 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-config\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.569045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.569020    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-serving-cert" pod "d5c77896-1ab6-4b63-9b09-377b9b7997de" (UID: "d5c77896-1ab6-4b63-9b09-377b9b7997de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:40.569137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.569049    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-oauth-config" pod "d5c77896-1ab6-4b63-9b09-377b9b7997de" (UID: "d5c77896-1ab6-4b63-9b09-377b9b7997de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:40.569193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.569164    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c77896-1ab6-4b63-9b09-377b9b7997de-kube-api-access-j74db" pod "d5c77896-1ab6-4b63-9b09-377b9b7997de" (UID: "d5c77896-1ab6-4b63-9b09-377b9b7997de"). InnerVolumeSpecName "kube-api-access-j74db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:40.668292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.668263    4633 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-serving-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.668292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.668291    4633 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c77896-1ab6-4b63-9b09-377b9b7997de-oauth-serving-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.668470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.668301    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j74db\" (UniqueName: \"kubernetes.io/projected/d5c77896-1ab6-4b63-9b09-377b9b7997de-kube-api-access-j74db\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:40.668470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:40.668312    4633 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c77896-1ab6-4b63-9b09-377b9b7997de-console-oauth-config\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:41.157085 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.157057    4633 generic.go:356] "Generic (PLEG): container finished" podID="d5c77896-1ab6-4b63-9b09-377b9b7997de" containerID="bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940" exitCode=0
Oct 30 21:02:41.157499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.157092    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b755fddbb-qd8t4" event={"ID":"d5c77896-1ab6-4b63-9b09-377b9b7997de","Type":"ContainerDied","Data":"bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940"}
Oct 30 21:02:41.157499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.157109    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b755fddbb-qd8t4"
Oct 30 21:02:41.157499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.157128    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b755fddbb-qd8t4" event={"ID":"d5c77896-1ab6-4b63-9b09-377b9b7997de","Type":"ContainerDied","Data":"72bce9ce0fd6188330b2a5cf95069dbd9f39df37f37735ddc5902877d36a837f"}
Oct 30 21:02:41.157499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.157148    4633 scope.go:122] "RemoveContainer" containerID="bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940"
Oct 30 21:02:41.166305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.166286    4633 scope.go:122] "RemoveContainer" containerID="bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940"
Oct 30 21:02:41.166587 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:02:41.166570    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940\": container with ID starting with bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940 not found: ID does not exist" containerID="bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940"
Oct 30 21:02:41.166632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.166596    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940"} err="failed to get container status \"bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940\": rpc error: code = NotFound desc = could not find container \"bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940\": container with ID starting with bd4a3d19a82413363899f402f54583fce4c82eb9bc6ed3471543bb510f3db940 not found: ID does not exist"
Oct 30 21:02:41.177283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.177260    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b755fddbb-qd8t4"]
Oct 30 21:02:41.180685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.180648    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b755fddbb-qd8t4"]
Oct 30 21:02:41.788320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:41.788287    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c77896-1ab6-4b63-9b09-377b9b7997de" path="/var/lib/kubelet/pods/d5c77896-1ab6-4b63-9b09-377b9b7997de/volumes"
Oct 30 21:02:45.474822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.474789    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86cc886fbc-wrtlh"]
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475057    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1c66adeb-03be-476e-960e-2544b9605375" containerName="insights-gathering"
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475067    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c66adeb-03be-476e-960e-2544b9605375" containerName="insights-gathering"
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475085    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b029cd0b-f2be-438d-ac98-55a1e85eca2c" containerName="console"
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475091    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b029cd0b-f2be-438d-ac98-55a1e85eca2c" containerName="console"
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475097    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d5c77896-1ab6-4b63-9b09-377b9b7997de" containerName="console"
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475102    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c77896-1ab6-4b63-9b09-377b9b7997de" containerName="console"
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475150    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b029cd0b-f2be-438d-ac98-55a1e85eca2c" containerName="console"
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475156    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c77896-1ab6-4b63-9b09-377b9b7997de" containerName="console"
Oct 30 21:02:45.475263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.475164    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c66adeb-03be-476e-960e-2544b9605375" containerName="insights-gathering"
Oct 30 21:02:45.493248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.493221    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cc886fbc-wrtlh"]
Oct 30 21:02:45.493409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.493317    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.618506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.618472    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-tmp\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.618506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.618512    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4cg\" (UniqueName: \"kubernetes.io/projected/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-kube-api-access-7j4cg\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.618771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.618536    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-trusted-ca-bundle\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.618771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.618600    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-oauth-serving-cert\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.618771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.618752    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-oauth-config\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.618902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.618791    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-service-ca\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.618902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.618858    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-config\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.619000 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.618939    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-serving-cert\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.719818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.719775    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-service-ca\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.719832    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-config\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.719878    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-serving-cert\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.719902    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-tmp\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.720043    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4cg\" (UniqueName: \"kubernetes.io/projected/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-kube-api-access-7j4cg\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.720099    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-trusted-ca-bundle\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.720323    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-oauth-serving-cert\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.720378    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-tmp\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.720414    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-oauth-config\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.720596    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-config\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.720605    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-service-ca\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.720988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.720969    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-oauth-serving-cert\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.721247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.721224    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-trusted-ca-bundle\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.722693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.722634    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-serving-cert\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.722821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.722803    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-oauth-config\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.728489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.728443    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4cg\" (UniqueName: \"kubernetes.io/projected/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-kube-api-access-7j4cg\") pod \"console-86cc886fbc-wrtlh\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") " pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:45.815432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:45.815396    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:46.111772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:46.111659    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cc886fbc-wrtlh"]
Oct 30 21:02:46.115047 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:02:46.115013    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80494b19_0bcb_4b8c_8d9c_1cb5af570b3b.slice/crio-0f6cf918b083cb9d7617251dbb3bf964956047bb202cb5d21fd93180dfdc22c0 WatchSource:0}: Error finding container 0f6cf918b083cb9d7617251dbb3bf964956047bb202cb5d21fd93180dfdc22c0: Status 404 returned error can't find the container with id 0f6cf918b083cb9d7617251dbb3bf964956047bb202cb5d21fd93180dfdc22c0
Oct 30 21:02:46.177552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:46.177521    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cc886fbc-wrtlh" event={"ID":"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b","Type":"ContainerStarted","Data":"0f6cf918b083cb9d7617251dbb3bf964956047bb202cb5d21fd93180dfdc22c0"}
Oct 30 21:02:47.182629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:47.182594    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cc886fbc-wrtlh" event={"ID":"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b","Type":"ContainerStarted","Data":"bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b"}
Oct 30 21:02:47.201214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:47.201159    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86cc886fbc-wrtlh" podStartSLOduration=2.201144588 podStartE2EDuration="2.201144588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:47.201052431 +0000 UTC m=+75.550779827" watchObservedRunningTime="2026-10-30 21:02:47.201144588 +0000 UTC m=+75.550871984"
Oct 30 21:02:55.013622 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:02:55.013585    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:02:55.816413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:55.816384    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:55.816586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:55.816434    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:55.821274 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:55.821254    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:56.217775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:56.217748    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:02:56.293737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:02:56.293704    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c796f96cd-mcnl7"]
Oct 30 21:03:04.664855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.664817    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-575dc7b45-6zxcw"]
Oct 30 21:03:04.669154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.669130    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.670476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.670457    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-serving-cert\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.670581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.670491    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-trusted-ca-bundle\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.670581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.670521    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-oauth-serving-cert\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.670581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.670536    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-config\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.670715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.670691    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32d59fa0-5d3d-4af4-a277-89de2848ae47-tmp\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.670776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.670760    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-service-ca\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.670811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.670792    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-oauth-config\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.670849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.670820    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-762ql\" (UniqueName: \"kubernetes.io/projected/32d59fa0-5d3d-4af4-a277-89de2848ae47-kube-api-access-762ql\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.682523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.682497    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575dc7b45-6zxcw"]
Oct 30 21:03:04.771315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771275    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32d59fa0-5d3d-4af4-a277-89de2848ae47-tmp\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.771494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771325    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-service-ca\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.771494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771346    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-oauth-config\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.771494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771371    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-762ql\" (UniqueName: \"kubernetes.io/projected/32d59fa0-5d3d-4af4-a277-89de2848ae47-kube-api-access-762ql\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.771494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771402    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-serving-cert\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.771494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771437    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-trusted-ca-bundle\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.771494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771484    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-oauth-serving-cert\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.771810 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771505    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-config\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.771810 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.771786    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32d59fa0-5d3d-4af4-a277-89de2848ae47-tmp\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.772280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.772256    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-config\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.772462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.772401    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-oauth-serving-cert\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.772553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.772530    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-trusted-ca-bundle\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.772693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.772658    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32d59fa0-5d3d-4af4-a277-89de2848ae47-service-ca\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.774179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.774154    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-serving-cert\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.774474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.774455    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32d59fa0-5d3d-4af4-a277-89de2848ae47-console-oauth-config\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.779363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.779332    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-762ql\" (UniqueName: \"kubernetes.io/projected/32d59fa0-5d3d-4af4-a277-89de2848ae47-kube-api-access-762ql\") pod \"console-575dc7b45-6zxcw\" (UID: \"32d59fa0-5d3d-4af4-a277-89de2848ae47\") " pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:04.981535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:04.981446    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:05.257144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:05.257062    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575dc7b45-6zxcw"]
Oct 30 21:03:05.259923 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:03:05.259893    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d59fa0_5d3d_4af4_a277_89de2848ae47.slice/crio-3ae1a91068ace71238f763f0b0812c69a23a8f6939d43d9f28b59da03451d8ee WatchSource:0}: Error finding container 3ae1a91068ace71238f763f0b0812c69a23a8f6939d43d9f28b59da03451d8ee: Status 404 returned error can't find the container with id 3ae1a91068ace71238f763f0b0812c69a23a8f6939d43d9f28b59da03451d8ee
Oct 30 21:03:06.250583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:06.250545    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575dc7b45-6zxcw" event={"ID":"32d59fa0-5d3d-4af4-a277-89de2848ae47","Type":"ContainerStarted","Data":"8c31d392fc0a175e6c5657ae8d7d3e607d77b951403b5ee4fd497161550314b8"}
Oct 30 21:03:06.251002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:06.250593    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575dc7b45-6zxcw" event={"ID":"32d59fa0-5d3d-4af4-a277-89de2848ae47","Type":"ContainerStarted","Data":"3ae1a91068ace71238f763f0b0812c69a23a8f6939d43d9f28b59da03451d8ee"}
Oct 30 21:03:06.269243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:06.269193    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-575dc7b45-6zxcw" podStartSLOduration=2.26918009 podStartE2EDuration="2.26918009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:06.268336927 +0000 UTC m=+94.618064322" watchObservedRunningTime="2026-10-30 21:03:06.26918009 +0000 UTC m=+94.618907485"
Oct 30 21:03:07.565521 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.565482    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-bqkgp-68gbm"]
Oct 30 21:03:07.566053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.565812    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-bqkgp-68gbm" podUID="4409113b-2c8a-4c21-b881-c0fc9cb12089" containerName="insights-gathering" containerID="cri-o://a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f" gracePeriod=30
Oct 30 21:03:07.806623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.806598    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:03:07.887217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.887142    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4409113b-2c8a-4c21-b881-c0fc9cb12089-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4409113b-2c8a-4c21-b881-c0fc9cb12089-archives-path\") pod \"4409113b-2c8a-4c21-b881-c0fc9cb12089\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") "
Oct 30 21:03:07.887217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.887178    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4409113b-2c8a-4c21-b881-c0fc9cb12089-kube-api-access-nt5dl\" (UniqueName: \"kubernetes.io/projected/4409113b-2c8a-4c21-b881-c0fc9cb12089-kube-api-access-nt5dl\") pod \"4409113b-2c8a-4c21-b881-c0fc9cb12089\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") "
Oct 30 21:03:07.887389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.887229    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-trusted-ca-bundle\") pod \"4409113b-2c8a-4c21-b881-c0fc9cb12089\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") "
Oct 30 21:03:07.887389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.887306    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-service-ca-bundle\") pod \"4409113b-2c8a-4c21-b881-c0fc9cb12089\" (UID: \"4409113b-2c8a-4c21-b881-c0fc9cb12089\") "
Oct 30 21:03:07.887558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.887538    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4409113b-2c8a-4c21-b881-c0fc9cb12089-archives-path" pod "4409113b-2c8a-4c21-b881-c0fc9cb12089" (UID: "4409113b-2c8a-4c21-b881-c0fc9cb12089"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:07.887704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.887657    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-trusted-ca-bundle" pod "4409113b-2c8a-4c21-b881-c0fc9cb12089" (UID: "4409113b-2c8a-4c21-b881-c0fc9cb12089"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:07.887748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.887698    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-service-ca-bundle" pod "4409113b-2c8a-4c21-b881-c0fc9cb12089" (UID: "4409113b-2c8a-4c21-b881-c0fc9cb12089"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:07.889417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.889400    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4409113b-2c8a-4c21-b881-c0fc9cb12089-kube-api-access-nt5dl" pod "4409113b-2c8a-4c21-b881-c0fc9cb12089" (UID: "4409113b-2c8a-4c21-b881-c0fc9cb12089"). InnerVolumeSpecName "kube-api-access-nt5dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:07.987742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.987711    4633 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-service-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:07.987742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.987736    4633 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/4409113b-2c8a-4c21-b881-c0fc9cb12089-archives-path\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:07.987742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.987746    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5dl\" (UniqueName: \"kubernetes.io/projected/4409113b-2c8a-4c21-b881-c0fc9cb12089-kube-api-access-nt5dl\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:07.987958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:07.987756    4633 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4409113b-2c8a-4c21-b881-c0fc9cb12089-trusted-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:08.259054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.259016    4633 generic.go:356] "Generic (PLEG): container finished" podID="4409113b-2c8a-4c21-b881-c0fc9cb12089" containerID="a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f" exitCode=2
Oct 30 21:03:08.259054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.259057    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-bqkgp-68gbm"
Oct 30 21:03:08.259260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.259104    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bqkgp-68gbm" event={"ID":"4409113b-2c8a-4c21-b881-c0fc9cb12089","Type":"ContainerDied","Data":"a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f"}
Oct 30 21:03:08.259260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.259148    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-bqkgp-68gbm" event={"ID":"4409113b-2c8a-4c21-b881-c0fc9cb12089","Type":"ContainerDied","Data":"6b1a0954b3c3146305a4996cd75869430d8fd0911a0630cfe142fe362f93ec12"}
Oct 30 21:03:08.259260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.259167    4633 scope.go:122] "RemoveContainer" containerID="a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f"
Oct 30 21:03:08.267457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.267438    4633 scope.go:122] "RemoveContainer" containerID="a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f"
Oct 30 21:03:08.267747 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:03:08.267725    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f\": container with ID starting with a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f not found: ID does not exist" containerID="a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f"
Oct 30 21:03:08.267798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.267756    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f"} err="failed to get container status \"a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f\": rpc error: code = NotFound desc = could not find container \"a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f\": container with ID starting with a090c5bde3ef724b66e11a004cc62f6e261571e12d70b4cc6522eff6716edc7f not found: ID does not exist"
Oct 30 21:03:08.282305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.282271    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-bqkgp-68gbm"]
Oct 30 21:03:08.286307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:08.286283    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-bqkgp-68gbm"]
Oct 30 21:03:09.787204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:09.787173    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4409113b-2c8a-4c21-b881-c0fc9cb12089" path="/var/lib/kubelet/pods/4409113b-2c8a-4c21-b881-c0fc9cb12089/volumes"
Oct 30 21:03:14.981935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:14.981886    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:14.982357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:14.981960    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:14.986924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:14.986902    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:15.287447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:15.287370    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-575dc7b45-6zxcw"
Oct 30 21:03:15.331936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:15.331908    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cc886fbc-wrtlh"]
Oct 30 21:03:21.312996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.312961    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c796f96cd-mcnl7" podUID="835ad90d-1857-4ec2-80ad-6c20e612ba11" containerName="console" containerID="cri-o://bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f" gracePeriod=15
Oct 30 21:03:21.557850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.557824    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:03:21.573156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573094    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-config\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-config\") pod \"835ad90d-1857-4ec2-80ad-6c20e612ba11\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") "
Oct 30 21:03:21.573156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573134    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835ad90d-1857-4ec2-80ad-6c20e612ba11-kube-api-access-jl2zt\" (UniqueName: \"kubernetes.io/projected/835ad90d-1857-4ec2-80ad-6c20e612ba11-kube-api-access-jl2zt\") pod \"835ad90d-1857-4ec2-80ad-6c20e612ba11\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") "
Oct 30 21:03:21.573331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573166    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-oauth-serving-cert\") pod \"835ad90d-1857-4ec2-80ad-6c20e612ba11\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") "
Oct 30 21:03:21.573331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573225    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-service-ca\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-service-ca\") pod \"835ad90d-1857-4ec2-80ad-6c20e612ba11\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") "
Oct 30 21:03:21.573331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573251    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-oauth-config\") pod \"835ad90d-1857-4ec2-80ad-6c20e612ba11\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") "
Oct 30 21:03:21.573331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573282    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-serving-cert\") pod \"835ad90d-1857-4ec2-80ad-6c20e612ba11\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") "
Oct 30 21:03:21.573520 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573331    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-trusted-ca-bundle\") pod \"835ad90d-1857-4ec2-80ad-6c20e612ba11\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") "
Oct 30 21:03:21.573520 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573364    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/835ad90d-1857-4ec2-80ad-6c20e612ba11-tmp\" (UniqueName: \"kubernetes.io/empty-dir/835ad90d-1857-4ec2-80ad-6c20e612ba11-tmp\") pod \"835ad90d-1857-4ec2-80ad-6c20e612ba11\" (UID: \"835ad90d-1857-4ec2-80ad-6c20e612ba11\") "
Oct 30 21:03:21.573614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573543    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-oauth-serving-cert" pod "835ad90d-1857-4ec2-80ad-6c20e612ba11" (UID: "835ad90d-1857-4ec2-80ad-6c20e612ba11"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:21.573614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573552    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-config" pod "835ad90d-1857-4ec2-80ad-6c20e612ba11" (UID: "835ad90d-1857-4ec2-80ad-6c20e612ba11"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:21.573752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573614    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-service-ca" pod "835ad90d-1857-4ec2-80ad-6c20e612ba11" (UID: "835ad90d-1857-4ec2-80ad-6c20e612ba11"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:21.573752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573629    4633 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-config\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:21.573752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573645    4633 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-oauth-serving-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:21.573873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573791    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835ad90d-1857-4ec2-80ad-6c20e612ba11-tmp" pod "835ad90d-1857-4ec2-80ad-6c20e612ba11" (UID: "835ad90d-1857-4ec2-80ad-6c20e612ba11"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:21.573959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.573900    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-trusted-ca-bundle" pod "835ad90d-1857-4ec2-80ad-6c20e612ba11" (UID: "835ad90d-1857-4ec2-80ad-6c20e612ba11"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:21.575426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.575407    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-oauth-config" pod "835ad90d-1857-4ec2-80ad-6c20e612ba11" (UID: "835ad90d-1857-4ec2-80ad-6c20e612ba11"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:21.575498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.575451    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835ad90d-1857-4ec2-80ad-6c20e612ba11-kube-api-access-jl2zt" pod "835ad90d-1857-4ec2-80ad-6c20e612ba11" (UID: "835ad90d-1857-4ec2-80ad-6c20e612ba11"). InnerVolumeSpecName "kube-api-access-jl2zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:21.575552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.575535    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-serving-cert" pod "835ad90d-1857-4ec2-80ad-6c20e612ba11" (UID: "835ad90d-1857-4ec2-80ad-6c20e612ba11"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:21.674656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.674619    4633 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-trusted-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:21.674656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.674650    4633 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/835ad90d-1857-4ec2-80ad-6c20e612ba11-tmp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:21.674872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.674660    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2zt\" (UniqueName: \"kubernetes.io/projected/835ad90d-1857-4ec2-80ad-6c20e612ba11-kube-api-access-jl2zt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:21.674872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.674696    4633 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/835ad90d-1857-4ec2-80ad-6c20e612ba11-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:21.674872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.674705    4633 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-oauth-config\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:21.674872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:21.674714    4633 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/835ad90d-1857-4ec2-80ad-6c20e612ba11-console-serving-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:22.307157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.307123    4633 generic.go:356] "Generic (PLEG): container finished" podID="835ad90d-1857-4ec2-80ad-6c20e612ba11" containerID="bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f" exitCode=0
Oct 30 21:03:22.307343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.307188    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c796f96cd-mcnl7" event={"ID":"835ad90d-1857-4ec2-80ad-6c20e612ba11","Type":"ContainerDied","Data":"bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f"}
Oct 30 21:03:22.307343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.307251    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c796f96cd-mcnl7" event={"ID":"835ad90d-1857-4ec2-80ad-6c20e612ba11","Type":"ContainerDied","Data":"a976a4719f22d02d3c500563b596624d92dd918b756e076aa674bf0a335ed0b0"}
Oct 30 21:03:22.307343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.307189    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c796f96cd-mcnl7"
Oct 30 21:03:22.307500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.307270    4633 scope.go:122] "RemoveContainer" containerID="bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f"
Oct 30 21:03:22.315511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.315351    4633 scope.go:122] "RemoveContainer" containerID="bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f"
Oct 30 21:03:22.315888 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:03:22.315652    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f\": container with ID starting with bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f not found: ID does not exist" containerID="bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f"
Oct 30 21:03:22.315888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.315699    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f"} err="failed to get container status \"bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f\": rpc error: code = NotFound desc = could not find container \"bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f\": container with ID starting with bf02e059fc2948a438b1cfe3811223b9eba2df630ced2d2c9249f77830793e3f not found: ID does not exist"
Oct 30 21:03:22.324710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.324688    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c796f96cd-mcnl7"]
Oct 30 21:03:22.328328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:22.328306    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c796f96cd-mcnl7"]
Oct 30 21:03:23.788365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:23.788323    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835ad90d-1857-4ec2-80ad-6c20e612ba11" path="/var/lib/kubelet/pods/835ad90d-1857-4ec2-80ad-6c20e612ba11/volumes"
Oct 30 21:03:25.012458 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:03:25.012423    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:03:40.356720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.356661    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86cc886fbc-wrtlh" podUID="80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" containerName="console" containerID="cri-o://bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b" gracePeriod=15
Oct 30 21:03:40.604103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.604075    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:03:40.715131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715096    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-config\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-config\") pod \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") "
Oct 30 21:03:40.715331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715140    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-tmp\") pod \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") "
Oct 30 21:03:40.715331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715162    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-kube-api-access-7j4cg\" (UniqueName: \"kubernetes.io/projected/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-kube-api-access-7j4cg\") pod \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") "
Oct 30 21:03:40.715331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715179    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-trusted-ca-bundle\") pod \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") "
Oct 30 21:03:40.715331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715211    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-oauth-config\") pod \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") "
Oct 30 21:03:40.715331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715233    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-serving-cert\") pod \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") "
Oct 30 21:03:40.715331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715280    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-oauth-serving-cert\") pod \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") "
Oct 30 21:03:40.715331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715329    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-service-ca\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-service-ca\") pod \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\" (UID: \"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b\") "
Oct 30 21:03:40.715787 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715414    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-tmp" pod "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" (UID: "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:03:40.715787 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715500    4633 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-tmp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:40.715787 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715715    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-config" pod "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" (UID: "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:40.715927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715822    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-oauth-serving-cert" pod "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" (UID: "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:40.715927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715890    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-trusted-ca-bundle" pod "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" (UID: "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:40.716013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.715955    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-service-ca" pod "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" (UID: "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:03:40.717471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.717445    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-kube-api-access-7j4cg" pod "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" (UID: "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b"). InnerVolumeSpecName "kube-api-access-7j4cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:03:40.717572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.717524    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-oauth-config" pod "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" (UID: "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:40.717798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.717774    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-serving-cert" pod "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" (UID: "80494b19-0bcb-4b8c-8d9c-1cb5af570b3b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:03:40.816109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.816074    4633 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-oauth-serving-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:40.816109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.816102    4633 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:40.816109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.816112    4633 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-config\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:40.816322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.816121    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j4cg\" (UniqueName: \"kubernetes.io/projected/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-kube-api-access-7j4cg\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:40.816322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.816131    4633 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-trusted-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:40.816322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.816140    4633 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-oauth-config\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:40.816322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:40.816148    4633 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b-console-serving-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:03:41.375557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.375521    4633 generic.go:356] "Generic (PLEG): container finished" podID="80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" containerID="bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b" exitCode=0
Oct 30 21:03:41.375983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.375571    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cc886fbc-wrtlh"
Oct 30 21:03:41.375983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.375600    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cc886fbc-wrtlh" event={"ID":"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b","Type":"ContainerDied","Data":"bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b"}
Oct 30 21:03:41.375983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.375650    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cc886fbc-wrtlh" event={"ID":"80494b19-0bcb-4b8c-8d9c-1cb5af570b3b","Type":"ContainerDied","Data":"0f6cf918b083cb9d7617251dbb3bf964956047bb202cb5d21fd93180dfdc22c0"}
Oct 30 21:03:41.375983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.375695    4633 scope.go:122] "RemoveContainer" containerID="bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b"
Oct 30 21:03:41.384522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.384500    4633 scope.go:122] "RemoveContainer" containerID="bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b"
Oct 30 21:03:41.384903 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:03:41.384884    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b\": container with ID starting with bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b not found: ID does not exist" containerID="bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b"
Oct 30 21:03:41.384973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.384918    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b"} err="failed to get container status \"bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b\": rpc error: code = NotFound desc = could not find container \"bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b\": container with ID starting with bb612fda9248412a9789825829303cb9b2b02a87643bb6bb533c69224ab8228b not found: ID does not exist"
Oct 30 21:03:41.397472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.397427    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cc886fbc-wrtlh"]
Oct 30 21:03:41.401490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.401471    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86cc886fbc-wrtlh"]
Oct 30 21:03:41.792149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:03:41.792117    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" path="/var/lib/kubelet/pods/80494b19-0bcb-4b8c-8d9c-1cb5af570b3b/volumes"
Oct 30 21:03:55.012419 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:03:55.012331    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:04:25.012685 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:04:25.012613    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:04:55.012995 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:04:55.012958    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:05:23.919715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.919613    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"]
Oct 30 21:05:23.920094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.919994    4633 cpu_manager.go:414] "containerMap: removing container" podUID="4409113b-2c8a-4c21-b881-c0fc9cb12089" containerName="insights-gathering"
Oct 30 21:05:23.920094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.920013    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4409113b-2c8a-4c21-b881-c0fc9cb12089" containerName="insights-gathering"
Oct 30 21:05:23.920094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.920051    4633 cpu_manager.go:414] "containerMap: removing container" podUID="80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" containerName="console"
Oct 30 21:05:23.920094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.920059    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" containerName="console"
Oct 30 21:05:23.920094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.920082    4633 cpu_manager.go:414] "containerMap: removing container" podUID="835ad90d-1857-4ec2-80ad-6c20e612ba11" containerName="console"
Oct 30 21:05:23.920094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.920090    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835ad90d-1857-4ec2-80ad-6c20e612ba11" containerName="console"
Oct 30 21:05:23.920273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.920162    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="80494b19-0bcb-4b8c-8d9c-1cb5af570b3b" containerName="console"
Oct 30 21:05:23.920273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.920175    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="835ad90d-1857-4ec2-80ad-6c20e612ba11" containerName="console"
Oct 30 21:05:23.920273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.920183    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4409113b-2c8a-4c21-b881-c0fc9cb12089" containerName="insights-gathering"
Oct 30 21:05:23.923435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.923417    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:23.932603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.932577    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:05:23.932765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.932687    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:05:23.933109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.933088    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-c2fg9\""
Oct 30 21:05:23.944283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:23.944262    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"]
Oct 30 21:05:24.067809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.067758    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.067990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.067815    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.067990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.067969    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6md\" (UniqueName: \"kubernetes.io/projected/95af1760-9426-4411-97c4-19674d809ed7-kube-api-access-wl6md\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.133255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.133218    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"]
Oct 30 21:05:24.136364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.136349    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.152126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.152095    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-p8f65\""
Oct 30 21:05:24.153102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.153080    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"]
Oct 30 21:05:24.168463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.168432    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.168585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.168473    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.168585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.168536    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6md\" (UniqueName: \"kubernetes.io/projected/95af1760-9426-4411-97c4-19674d809ed7-kube-api-access-wl6md\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.168807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.168788    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.168889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.168824    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.180864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.180815    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6md\" (UniqueName: \"kubernetes.io/projected/95af1760-9426-4411-97c4-19674d809ed7-kube-api-access-wl6md\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.240197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.240168    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:05:24.268919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.268896    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.269029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.268925    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.269029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.268947    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dp2z\" (UniqueName: \"kubernetes.io/projected/0d2dec23-ac5c-4a58-8528-71411db5e8c0-kube-api-access-8dp2z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.370354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.370320    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.370354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.370354    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.370611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.370375    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dp2z\" (UniqueName: \"kubernetes.io/projected/0d2dec23-ac5c-4a58-8528-71411db5e8c0-kube-api-access-8dp2z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.370794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.370772    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.370858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.370814    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.380199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.380173    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dp2z\" (UniqueName: \"kubernetes.io/projected/0d2dec23-ac5c-4a58-8528-71411db5e8c0-kube-api-access-8dp2z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.451122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.451049    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:05:24.521786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.521744    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"]
Oct 30 21:05:24.526903 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:24.526872    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95af1760_9426_4411_97c4_19674d809ed7.slice/crio-259fde41af621b4ed1af180d07dfe4af99d31a3e93828b16acf3ca536b7191dc WatchSource:0}: Error finding container 259fde41af621b4ed1af180d07dfe4af99d31a3e93828b16acf3ca536b7191dc: Status 404 returned error can't find the container with id 259fde41af621b4ed1af180d07dfe4af99d31a3e93828b16acf3ca536b7191dc
Oct 30 21:05:24.705616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.705544    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v" event={"ID":"95af1760-9426-4411-97c4-19674d809ed7","Type":"ContainerStarted","Data":"259fde41af621b4ed1af180d07dfe4af99d31a3e93828b16acf3ca536b7191dc"}
Oct 30 21:05:24.737141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:24.737104    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"]
Oct 30 21:05:24.739838 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:24.739810    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2dec23_ac5c_4a58_8528_71411db5e8c0.slice/crio-535f314a4254a0cc0dd3ca6968964b50a195b122b42e481581bdb1e6cb4cad25 WatchSource:0}: Error finding container 535f314a4254a0cc0dd3ca6968964b50a195b122b42e481581bdb1e6cb4cad25: Status 404 returned error can't find the container with id 535f314a4254a0cc0dd3ca6968964b50a195b122b42e481581bdb1e6cb4cad25
Oct 30 21:05:25.013847 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:05:25.013765    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:05:25.118813 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.118759    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"]
Oct 30 21:05:25.123350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.123330    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.130457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.130431    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-mzlhf\""
Oct 30 21:05:25.135068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.134664    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"]
Oct 30 21:05:25.178012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.177976    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rk8\" (UniqueName: \"kubernetes.io/projected/367837ae-fcd7-44d3-a5d7-012d73bec03b-kube-api-access-p9rk8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.178012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.178011    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.178263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.178039    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.279508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.279428    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.279508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.279503    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rk8\" (UniqueName: \"kubernetes.io/projected/367837ae-fcd7-44d3-a5d7-012d73bec03b-kube-api-access-p9rk8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.279782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.279534    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.279938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.279917    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.279995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.279945    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.289199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.289172    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rk8\" (UniqueName: \"kubernetes.io/projected/367837ae-fcd7-44d3-a5d7-012d73bec03b-kube-api-access-p9rk8\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.440256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.440214    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:05:25.712396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.712356    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj" event={"ID":"0d2dec23-ac5c-4a58-8528-71411db5e8c0","Type":"ContainerStarted","Data":"535f314a4254a0cc0dd3ca6968964b50a195b122b42e481581bdb1e6cb4cad25"}
Oct 30 21:05:25.733430 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.733399    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"]
Oct 30 21:05:25.737911 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:25.737869    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367837ae_fcd7_44d3_a5d7_012d73bec03b.slice/crio-e85ad4f52589fe61d039613baa681929a62cdd80e4b19cec6a70ae4eefec637f WatchSource:0}: Error finding container e85ad4f52589fe61d039613baa681929a62cdd80e4b19cec6a70ae4eefec637f: Status 404 returned error can't find the container with id e85ad4f52589fe61d039613baa681929a62cdd80e4b19cec6a70ae4eefec637f
Oct 30 21:05:25.912935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.912896    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"]
Oct 30 21:05:25.917470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.917441    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:25.928532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.928510    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-thf68\""
Oct 30 21:05:25.933689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.933608    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"]
Oct 30 21:05:25.987215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.987127    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:25.987389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.987267    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rdp\" (UniqueName: \"kubernetes.io/projected/2cf875be-7347-490d-9f81-af2730212752-kube-api-access-f9rdp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:25.987465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:25.987385    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:26.018559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.018533    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"]
Oct 30 21:05:26.023101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.023079    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.028538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.028511    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-p9qz2\""
Oct 30 21:05:26.035394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.035333    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"]
Oct 30 21:05:26.088470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.088433    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:26.088645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.088513    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fldf\" (UniqueName: \"kubernetes.io/projected/813575cb-0644-4a95-9881-caba2b383d98-kube-api-access-9fldf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.088645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.088547    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.088645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.088576    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:26.088645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.088616    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rdp\" (UniqueName: \"kubernetes.io/projected/2cf875be-7347-490d-9f81-af2730212752-kube-api-access-f9rdp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:26.088897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.088647    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.089082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.089060    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:26.089312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.089294    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:26.104255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.104207    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rdp\" (UniqueName: \"kubernetes.io/projected/2cf875be-7347-490d-9f81-af2730212752-kube-api-access-f9rdp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:26.117580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.117554    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"]
Oct 30 21:05:26.121392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.121373    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.125853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.125348    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-pqkx7\""
Oct 30 21:05:26.139154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.139132    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"]
Oct 30 21:05:26.189419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.189383    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.189582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.189436    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt5jg\" (UniqueName: \"kubernetes.io/projected/6f2137c2-d968-4045-8caf-bec6fa171bc2-kube-api-access-rt5jg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.189582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.189464    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.189582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.189497    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fldf\" (UniqueName: \"kubernetes.io/projected/813575cb-0644-4a95-9881-caba2b383d98-kube-api-access-9fldf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.189582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.189534    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.189857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.189603    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.190057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.190035    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.190791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.190600    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.199007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.198980    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fldf\" (UniqueName: \"kubernetes.io/projected/813575cb-0644-4a95-9881-caba2b383d98-kube-api-access-9fldf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.213200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.213173    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"]
Oct 30 21:05:26.216529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.216509    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.220410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.220386    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-rbrlp\""
Oct 30 21:05:26.231198 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.231172    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"]
Oct 30 21:05:26.242696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.236870    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:05:26.291944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.291901    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45vv9\" (UniqueName: \"kubernetes.io/projected/169245ae-3643-4bfa-b8b3-f5b5e5308d51-kube-api-access-45vv9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.292096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.291970    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.292096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.292015    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.292096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.292048    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.292096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.292077    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt5jg\" (UniqueName: \"kubernetes.io/projected/6f2137c2-d968-4045-8caf-bec6fa171bc2-kube-api-access-rt5jg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.292226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.292106    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.296454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.296201    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.296454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.296401    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.306445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.306169    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt5jg\" (UniqueName: \"kubernetes.io/projected/6f2137c2-d968-4045-8caf-bec6fa171bc2-kube-api-access-rt5jg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.345054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.345020    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:05:26.393090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.393053    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45vv9\" (UniqueName: \"kubernetes.io/projected/169245ae-3643-4bfa-b8b3-f5b5e5308d51-kube-api-access-45vv9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.393265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.393104    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.393265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.393128    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.393556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.393529    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.394068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.394045    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.405154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.405131    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45vv9\" (UniqueName: \"kubernetes.io/projected/169245ae-3643-4bfa-b8b3-f5b5e5308d51-kube-api-access-45vv9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.442718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.442648    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:05:26.539378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.539344    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"]
Oct 30 21:05:26.541457 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:26.541410    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf875be_7347_490d_9f81_af2730212752.slice/crio-eabb602d77c97dbb98d5ceeee558abf0b93a36910d9bc406a6bece8915080298 WatchSource:0}: Error finding container eabb602d77c97dbb98d5ceeee558abf0b93a36910d9bc406a6bece8915080298: Status 404 returned error can't find the container with id eabb602d77c97dbb98d5ceeee558abf0b93a36910d9bc406a6bece8915080298
Oct 30 21:05:26.543027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.542561    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:05:26.632443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.632317    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"]
Oct 30 21:05:26.639389 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:26.639351    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813575cb_0644_4a95_9881_caba2b383d98.slice/crio-edb3f99c0d441fa7493ef57e912cbdf8ae2f5f10fe42ef712cb466c2293aa84e WatchSource:0}: Error finding container edb3f99c0d441fa7493ef57e912cbdf8ae2f5f10fe42ef712cb466c2293aa84e: Status 404 returned error can't find the container with id edb3f99c0d441fa7493ef57e912cbdf8ae2f5f10fe42ef712cb466c2293aa84e
Oct 30 21:05:26.722504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.722466    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6" event={"ID":"367837ae-fcd7-44d3-a5d7-012d73bec03b","Type":"ContainerStarted","Data":"e85ad4f52589fe61d039613baa681929a62cdd80e4b19cec6a70ae4eefec637f"}
Oct 30 21:05:26.724298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.724234    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k" event={"ID":"813575cb-0644-4a95-9881-caba2b383d98","Type":"ContainerStarted","Data":"edb3f99c0d441fa7493ef57e912cbdf8ae2f5f10fe42ef712cb466c2293aa84e"}
Oct 30 21:05:26.726171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.726143    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9" event={"ID":"2cf875be-7347-490d-9f81-af2730212752","Type":"ContainerStarted","Data":"eabb602d77c97dbb98d5ceeee558abf0b93a36910d9bc406a6bece8915080298"}
Oct 30 21:05:26.741799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.741698    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"]
Oct 30 21:05:26.830771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:26.830687    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"]
Oct 30 21:05:26.835515 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:26.835485    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169245ae_3643_4bfa_b8b3_f5b5e5308d51.slice/crio-4c51388418591b6ef2fc8c860d02356bddd59f4460bdbea72c6b17284c5e31bf WatchSource:0}: Error finding container 4c51388418591b6ef2fc8c860d02356bddd59f4460bdbea72c6b17284c5e31bf: Status 404 returned error can't find the container with id 4c51388418591b6ef2fc8c860d02356bddd59f4460bdbea72c6b17284c5e31bf
Oct 30 21:05:27.739661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:27.739620    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx" event={"ID":"6f2137c2-d968-4045-8caf-bec6fa171bc2","Type":"ContainerStarted","Data":"505f5decbbd3a631210d12a42dedf41a99bbd54c5a5ddc58320876c1b1a0cb74"}
Oct 30 21:05:27.743939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:27.743907    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76" event={"ID":"169245ae-3643-4bfa-b8b3-f5b5e5308d51","Type":"ContainerStarted","Data":"4c51388418591b6ef2fc8c860d02356bddd59f4460bdbea72c6b17284c5e31bf"}
Oct 30 21:05:28.295014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.293371    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"]
Oct 30 21:05:28.298856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.297733    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.311637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.311614    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:05:28.311803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.311776    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-l56fx\""
Oct 30 21:05:28.312336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.311865    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:05:28.312336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.311927    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:05:28.327825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.327747    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"]
Oct 30 21:05:28.413133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413098    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvwp\" (UniqueName: \"kubernetes.io/projected/bfa29907-7f35-4ba2-8303-54079a98cb5c-kube-api-access-hqvwp\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.413311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413159    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa29907-7f35-4ba2-8303-54079a98cb5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.413311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413197    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.413311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413229    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.413311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413259    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.413311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413285    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.413311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413306    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.413641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413339    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.413641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.413367    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.431901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.430797    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"]
Oct 30 21:05:28.436072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.435449    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.455694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.455591    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"]
Oct 30 21:05:28.464283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.464241    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-jzfh8\""
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.513951    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514017    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514100    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514135    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514183    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514211    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514239    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514278    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514303    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514328    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514351    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514384    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514421    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514480    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvwp\" (UniqueName: \"kubernetes.io/projected/bfa29907-7f35-4ba2-8303-54079a98cb5c-kube-api-access-hqvwp\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514542    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmqz\" (UniqueName: \"kubernetes.io/projected/9ac64e06-5eef-43d8-94e5-64698a0d7e52-kube-api-access-5tmqz\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514583    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa29907-7f35-4ba2-8303-54079a98cb5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514615    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.514649    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.515171    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.515513    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.515983    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.517178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.516194    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.519214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.519193    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa29907-7f35-4ba2-8303-54079a98cb5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.523071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.522466    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.527458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.527410    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvwp\" (UniqueName: \"kubernetes.io/projected/bfa29907-7f35-4ba2-8303-54079a98cb5c-kube-api-access-hqvwp\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.528757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.528731    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.538187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.538004    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa29907-7f35-4ba2-8303-54079a98cb5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-msql6\" (UID: \"bfa29907-7f35-4ba2-8303-54079a98cb5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615389    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615435    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615461    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615489    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615558    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmqz\" (UniqueName: \"kubernetes.io/projected/9ac64e06-5eef-43d8-94e5-64698a0d7e52-kube-api-access-5tmqz\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615589    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615619    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615689    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.615731    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.616160    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.617549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.616401    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.621755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.618592    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.621755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.618830    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.621755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.619390    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.621755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.619541    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:28.621755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.621372    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.623840 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.622953    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.626845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.626097    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ac64e06-5eef-43d8-94e5-64698a0d7e52-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.635259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.635166    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmqz\" (UniqueName: \"kubernetes.io/projected/9ac64e06-5eef-43d8-94e5-64698a0d7e52-kube-api-access-5tmqz\") pod \"openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g\" (UID: \"9ac64e06-5eef-43d8-94e5-64698a0d7e52\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.757397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.757083    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:28.979622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:28.979588    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"]
Oct 30 21:05:28.986527 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:28.986467    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa29907_7f35_4ba2_8303_54079a98cb5c.slice/crio-74a96441851afc3051973d8c25d914e2db459a5656da08d22709da930b0bfd02 WatchSource:0}: Error finding container 74a96441851afc3051973d8c25d914e2db459a5656da08d22709da930b0bfd02: Status 404 returned error can't find the container with id 74a96441851afc3051973d8c25d914e2db459a5656da08d22709da930b0bfd02
Oct 30 21:05:29.090968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:29.090938    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"]
Oct 30 21:05:29.775150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:29.775109    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g" event={"ID":"9ac64e06-5eef-43d8-94e5-64698a0d7e52","Type":"ContainerStarted","Data":"6633feacf4eecf11986187ff7b50f330ff7a23772dc9e197c1f8de4bc14f2a00"}
Oct 30 21:05:29.776802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:29.776722    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" event={"ID":"bfa29907-7f35-4ba2-8303-54079a98cb5c","Type":"ContainerStarted","Data":"74a96441851afc3051973d8c25d914e2db459a5656da08d22709da930b0bfd02"}
Oct 30 21:05:30.707273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.707233    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"]
Oct 30 21:05:30.711685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.711647    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.721387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.721353    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"]
Oct 30 21:05:30.752651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.750815    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"]
Oct 30 21:05:30.755395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.754964    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.772213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.772139    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"]
Oct 30 21:05:30.802102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.802066    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"]
Oct 30 21:05:30.806852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.806826    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:30.822197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.820104    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"]
Oct 30 21:05:30.839065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.839033    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"]
Oct 30 21:05:30.841436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.841012    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48xb\" (UniqueName: \"kubernetes.io/projected/b369efab-35eb-48aa-9f71-9900ee02f727-kube-api-access-w48xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.841436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.841120    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.841436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.841156    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.841436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.841182    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rbc\" (UniqueName: \"kubernetes.io/projected/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-kube-api-access-27rbc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.841436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.841214    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.841436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.841241    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.843842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.843819    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:30.861811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.859908    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"]
Oct 30 21:05:30.941951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.941913    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnd7j\" (UniqueName: \"kubernetes.io/projected/96bb663c-3225-4595-83e7-82b322868b20-kube-api-access-jnd7j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:30.942144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.941969    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48xb\" (UniqueName: \"kubernetes.io/projected/b369efab-35eb-48aa-9f71-9900ee02f727-kube-api-access-w48xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.942144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942007    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6kpc\" (UniqueName: \"kubernetes.io/projected/6e527a0b-1fa4-402c-ab56-6dd27aeca974-kube-api-access-w6kpc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:30.942144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942067    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:30.942144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942141    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:30.942378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942166    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:30.942378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942197    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:30.942378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942280    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.942378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942315    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.942378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942348    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rbc\" (UniqueName: \"kubernetes.io/projected/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-kube-api-access-27rbc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.942615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942381    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.942615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942409    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.942948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.942925    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.943835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.943805    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.944454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.944428    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.944454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.944447    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:30.964697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.963958    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48xb\" (UniqueName: \"kubernetes.io/projected/b369efab-35eb-48aa-9f71-9900ee02f727-kube-api-access-w48xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:30.964697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:30.964584    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rbc\" (UniqueName: \"kubernetes.io/projected/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-kube-api-access-27rbc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:31.043731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.042524    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:05:31.043731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.043166    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnd7j\" (UniqueName: \"kubernetes.io/projected/96bb663c-3225-4595-83e7-82b322868b20-kube-api-access-jnd7j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:31.043731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.043206    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6kpc\" (UniqueName: \"kubernetes.io/projected/6e527a0b-1fa4-402c-ab56-6dd27aeca974-kube-api-access-w6kpc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:31.043731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.043255    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:31.043731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.043289    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:31.043731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.043314    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:31.043731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.043341    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:31.044325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.044271    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:31.044887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.044708    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:31.044887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.044844    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:31.044996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.044939    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:31.075166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.075132    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnd7j\" (UniqueName: \"kubernetes.io/projected/96bb663c-3225-4595-83e7-82b322868b20-kube-api-access-jnd7j\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:31.076085 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.076060    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6kpc\" (UniqueName: \"kubernetes.io/projected/6e527a0b-1fa4-402c-ab56-6dd27aeca974-kube-api-access-w6kpc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:31.082927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.082899    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:05:31.136929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.136897    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:05:31.174355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.174317    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:05:31.411629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.411600    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"]
Oct 30 21:05:31.469413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.468710    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"]
Oct 30 21:05:31.485600 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:31.485543    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb369efab_35eb_48aa_9f71_9900ee02f727.slice/crio-64667d88555ff6ff31d09e0b6e3215d6f3e23f1bdf9e06bed7af6ebf30419e3b WatchSource:0}: Error finding container 64667d88555ff6ff31d09e0b6e3215d6f3e23f1bdf9e06bed7af6ebf30419e3b: Status 404 returned error can't find the container with id 64667d88555ff6ff31d09e0b6e3215d6f3e23f1bdf9e06bed7af6ebf30419e3b
Oct 30 21:05:31.492701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.492645    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"]
Oct 30 21:05:31.540380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.540299    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"]
Oct 30 21:05:31.548742 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:05:31.548708    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bb663c_3225_4595_83e7_82b322868b20.slice/crio-e2374a4e510cbab2856ef33b8e9847f805bf1e4395e3f6d6b36c87880420b06f WatchSource:0}: Error finding container e2374a4e510cbab2856ef33b8e9847f805bf1e4395e3f6d6b36c87880420b06f: Status 404 returned error can't find the container with id e2374a4e510cbab2856ef33b8e9847f805bf1e4395e3f6d6b36c87880420b06f
Oct 30 21:05:31.821083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.821000    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc" event={"ID":"b369efab-35eb-48aa-9f71-9900ee02f727","Type":"ContainerStarted","Data":"64667d88555ff6ff31d09e0b6e3215d6f3e23f1bdf9e06bed7af6ebf30419e3b"}
Oct 30 21:05:31.855103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.855058    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k" event={"ID":"96bb663c-3225-4595-83e7-82b322868b20","Type":"ContainerStarted","Data":"e2374a4e510cbab2856ef33b8e9847f805bf1e4395e3f6d6b36c87880420b06f"}
Oct 30 21:05:31.874078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.874040    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz" event={"ID":"6e527a0b-1fa4-402c-ab56-6dd27aeca974","Type":"ContainerStarted","Data":"98bdb3c0a27d10f65ccf07d20a2d17d16a4513b3250fdb33b0992c05278bf420"}
Oct 30 21:05:31.888320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:31.888283    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl" event={"ID":"0f495ab7-b401-41de-9b30-af7a3f5b6bb5","Type":"ContainerStarted","Data":"673f9f7bf0267714e4ec19ea49c6d9909cfa0bf31db0fc169943bdc8c566d9bf"}
Oct 30 21:05:41.329748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:41.329003    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:05:41.329748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:41.329095    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:05:41.329748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:41.329140    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:05:41.764574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:41.764529    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:05:41.764746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:41.764721    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:05:41.764838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:41.764777    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:05:41.999633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:41.998572    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" event={"ID":"bfa29907-7f35-4ba2-8303-54079a98cb5c","Type":"ContainerStarted","Data":"ab9a933dd20e65f913f3c659f394f70598a7d75bfed16a6d432db304a8e68ba5"}
Oct 30 21:05:42.004424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:42.003006    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g" event={"ID":"9ac64e06-5eef-43d8-94e5-64698a0d7e52","Type":"ContainerStarted","Data":"98dfe057a779b89fb6753bd09cc2fac4ecc94d150d6c5681fd7f5836d5f37760"}
Oct 30 21:05:42.023746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:42.023087    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podStartSLOduration=1.688233901 podStartE2EDuration="14.023072705s" totalImagesPullingTime="12.334838804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:42.022379667 +0000 UTC m=+250.372107065" watchObservedRunningTime="2026-10-30 21:05:42.023072705 +0000 UTC m=+250.372800101"
Oct 30 21:05:42.048626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:42.047926    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g" podStartSLOduration=1.385482925 podStartE2EDuration="14.047903617s" totalImagesPullingTime="12.662420692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:42.045618192 +0000 UTC m=+250.395345590" watchObservedRunningTime="2026-10-30 21:05:42.047903617 +0000 UTC m=+250.397631010"
Oct 30 21:05:42.622479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:42.621061    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:42.627439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:42.627411    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:42.758578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:42.758540    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:42.765203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:42.765177    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:43.021492 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:43.021406    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:43.021492 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:43.021466    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:43.022626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:43.022600    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g"
Oct 30 21:05:43.023547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:43.023527    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:05:45.034511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:45.034003    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v" event={"ID":"95af1760-9426-4411-97c4-19674d809ed7","Type":"ContainerStarted","Data":"107a82ab9eda88e549df4faef93143a495e8d7c484b95b844409b454dfca55ef"}
Oct 30 21:05:45.043205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:45.041435    4633 generic.go:356] "Generic (PLEG): container finished" podID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerID="d9d5a0a83ec9c3c493d772c524e5cd57cb644cc4f610b4297901e035cc44ce0d" exitCode=0
Oct 30 21:05:45.043205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:45.042633    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj" event={"ID":"0d2dec23-ac5c-4a58-8528-71411db5e8c0","Type":"ContainerDied","Data":"d9d5a0a83ec9c3c493d772c524e5cd57cb644cc4f610b4297901e035cc44ce0d"}
Oct 30 21:05:46.049349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:46.049313    4633 generic.go:356] "Generic (PLEG): container finished" podID="96bb663c-3225-4595-83e7-82b322868b20" containerID="3d1a932206d9074cbff69d8888f249297a82d02a2e40019ca31d622f358ce26f" exitCode=0
Oct 30 21:05:46.049796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:46.049406    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k" event={"ID":"96bb663c-3225-4595-83e7-82b322868b20","Type":"ContainerDied","Data":"3d1a932206d9074cbff69d8888f249297a82d02a2e40019ca31d622f358ce26f"}
Oct 30 21:05:46.055563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:46.055539    4633 generic.go:356] "Generic (PLEG): container finished" podID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerID="dbd3b7e2b6f6b3cbd2aa3a282fbb13a59e9d7fa6ea5492e9f3c0031fd1756a5c" exitCode=0
Oct 30 21:05:46.055714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:46.055661    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl" event={"ID":"0f495ab7-b401-41de-9b30-af7a3f5b6bb5","Type":"ContainerDied","Data":"dbd3b7e2b6f6b3cbd2aa3a282fbb13a59e9d7fa6ea5492e9f3c0031fd1756a5c"}
Oct 30 21:05:46.058690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:46.058088    4633 generic.go:356] "Generic (PLEG): container finished" podID="95af1760-9426-4411-97c4-19674d809ed7" containerID="107a82ab9eda88e549df4faef93143a495e8d7c484b95b844409b454dfca55ef" exitCode=0
Oct 30 21:05:46.058690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:46.058157    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v" event={"ID":"95af1760-9426-4411-97c4-19674d809ed7","Type":"ContainerDied","Data":"107a82ab9eda88e549df4faef93143a495e8d7c484b95b844409b454dfca55ef"}
Oct 30 21:05:47.065283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:47.065192    4633 generic.go:356] "Generic (PLEG): container finished" podID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerID="e285ef7fef164e5b2d36e5140c3597e4425ffb999e00f077758ef0793bc6cc79" exitCode=0
Oct 30 21:05:47.065738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:47.065302    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz" event={"ID":"6e527a0b-1fa4-402c-ab56-6dd27aeca974","Type":"ContainerDied","Data":"e285ef7fef164e5b2d36e5140c3597e4425ffb999e00f077758ef0793bc6cc79"}
Oct 30 21:05:47.067450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:47.067426    4633 generic.go:356] "Generic (PLEG): container finished" podID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerID="17ee0bec0d295e17c0054644ea3a01a575dc115fc40766bf757bda39e1b4440f" exitCode=0
Oct 30 21:05:47.067576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:47.067549    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6" event={"ID":"367837ae-fcd7-44d3-a5d7-012d73bec03b","Type":"ContainerDied","Data":"17ee0bec0d295e17c0054644ea3a01a575dc115fc40766bf757bda39e1b4440f"}
Oct 30 21:05:47.069474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:47.069448    4633 generic.go:356] "Generic (PLEG): container finished" podID="b369efab-35eb-48aa-9f71-9900ee02f727" containerID="3f04240112251dc488383a8d476e85076966b18c4e410bb2f876c32f742c238d" exitCode=0
Oct 30 21:05:47.069587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:47.069524    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc" event={"ID":"b369efab-35eb-48aa-9f71-9900ee02f727","Type":"ContainerDied","Data":"3f04240112251dc488383a8d476e85076966b18c4e410bb2f876c32f742c238d"}
Oct 30 21:05:48.076442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:48.076409    4633 generic.go:356] "Generic (PLEG): container finished" podID="2cf875be-7347-490d-9f81-af2730212752" containerID="23faa5b88e8bf940a88ef5be331461386417075545e586250e83754a75f784f3" exitCode=0
Oct 30 21:05:48.076916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:48.076498    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9" event={"ID":"2cf875be-7347-490d-9f81-af2730212752","Type":"ContainerDied","Data":"23faa5b88e8bf940a88ef5be331461386417075545e586250e83754a75f784f3"}
Oct 30 21:05:49.091495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.091458    4633 generic.go:356] "Generic (PLEG): container finished" podID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerID="c5bd3b13c4a079239f8391b32945e904c69d80ef7ec22a31f60dd6e0a6df8187" exitCode=0
Oct 30 21:05:49.091996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.091569    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx" event={"ID":"6f2137c2-d968-4045-8caf-bec6fa171bc2","Type":"ContainerDied","Data":"c5bd3b13c4a079239f8391b32945e904c69d80ef7ec22a31f60dd6e0a6df8187"}
Oct 30 21:05:49.094065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.094040    4633 generic.go:356] "Generic (PLEG): container finished" podID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerID="eaeb28916f5843a31d31b88de4cd90f367f46762e93a00fc52c857e20cae6672" exitCode=0
Oct 30 21:05:49.094196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.094128    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6" event={"ID":"367837ae-fcd7-44d3-a5d7-012d73bec03b","Type":"ContainerDied","Data":"eaeb28916f5843a31d31b88de4cd90f367f46762e93a00fc52c857e20cae6672"}
Oct 30 21:05:49.096745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.096653    4633 generic.go:356] "Generic (PLEG): container finished" podID="813575cb-0644-4a95-9881-caba2b383d98" containerID="530a174d649d8e54a5da489093794ae8fcd71b06a6c9084bcbf9fa244ff212ca" exitCode=0
Oct 30 21:05:49.096816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.096749    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k" event={"ID":"813575cb-0644-4a95-9881-caba2b383d98","Type":"ContainerDied","Data":"530a174d649d8e54a5da489093794ae8fcd71b06a6c9084bcbf9fa244ff212ca"}
Oct 30 21:05:49.098606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.098579    4633 generic.go:356] "Generic (PLEG): container finished" podID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerID="e72715d4a6128dabd5f88d683d6f26c2b037cb012168009c99dea222dad5791c" exitCode=0
Oct 30 21:05:49.098726 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.098706    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76" event={"ID":"169245ae-3643-4bfa-b8b3-f5b5e5308d51","Type":"ContainerDied","Data":"e72715d4a6128dabd5f88d683d6f26c2b037cb012168009c99dea222dad5791c"}
Oct 30 21:05:49.102971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.102066    4633 generic.go:356] "Generic (PLEG): container finished" podID="96bb663c-3225-4595-83e7-82b322868b20" containerID="6f5c72df65a40429ef203f1c80d351d5a9cff31d47739597bc177198a0dfc3c6" exitCode=0
Oct 30 21:05:49.102971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.102147    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k" event={"ID":"96bb663c-3225-4595-83e7-82b322868b20","Type":"ContainerDied","Data":"6f5c72df65a40429ef203f1c80d351d5a9cff31d47739597bc177198a0dfc3c6"}
Oct 30 21:05:49.104638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.104616    4633 generic.go:356] "Generic (PLEG): container finished" podID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerID="7945c49dd243b231722132bd7b0fa2b5cebb1c900ae944c670c1b1da8b647909" exitCode=0
Oct 30 21:05:49.104756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:49.104685    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl" event={"ID":"0f495ab7-b401-41de-9b30-af7a3f5b6bb5","Type":"ContainerDied","Data":"7945c49dd243b231722132bd7b0fa2b5cebb1c900ae944c670c1b1da8b647909"}
Oct 30 21:05:50.114007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.113903    4633 generic.go:356] "Generic (PLEG): container finished" podID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerID="40be83990cc4eb9ed7dd1d9ad3467f686ef3b6ec20fb65e2c41a5ee686575ed2" exitCode=0
Oct 30 21:05:50.114007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.113987    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76" event={"ID":"169245ae-3643-4bfa-b8b3-f5b5e5308d51","Type":"ContainerDied","Data":"40be83990cc4eb9ed7dd1d9ad3467f686ef3b6ec20fb65e2c41a5ee686575ed2"}
Oct 30 21:05:50.122686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.121931    4633 generic.go:356] "Generic (PLEG): container finished" podID="b369efab-35eb-48aa-9f71-9900ee02f727" containerID="a87dbbeb12b40ce4080b361b88da935f83ee90c752725a9c53e1771fbf636fc3" exitCode=0
Oct 30 21:05:50.122686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.122036    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc" event={"ID":"b369efab-35eb-48aa-9f71-9900ee02f727","Type":"ContainerDied","Data":"a87dbbeb12b40ce4080b361b88da935f83ee90c752725a9c53e1771fbf636fc3"}
Oct 30 21:05:50.130942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.130910    4633 generic.go:356] "Generic (PLEG): container finished" podID="2cf875be-7347-490d-9f81-af2730212752" containerID="23028b4761cea681f8c98db3a0d6fa1ee3abde6931f03c42c4a9ec001f6dfd11" exitCode=0
Oct 30 21:05:50.131598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.131544    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9" event={"ID":"2cf875be-7347-490d-9f81-af2730212752","Type":"ContainerDied","Data":"23028b4761cea681f8c98db3a0d6fa1ee3abde6931f03c42c4a9ec001f6dfd11"}
Oct 30 21:05:50.139372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.139319    4633 generic.go:356] "Generic (PLEG): container finished" podID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerID="c61622c61b42ca3096a7b19f672edc4120fa7ded48598ba20ab7cfdb5485e154" exitCode=0
Oct 30 21:05:50.139481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.139400    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz" event={"ID":"6e527a0b-1fa4-402c-ab56-6dd27aeca974","Type":"ContainerDied","Data":"c61622c61b42ca3096a7b19f672edc4120fa7ded48598ba20ab7cfdb5485e154"}
Oct 30 21:05:50.144209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.144185    4633 generic.go:356] "Generic (PLEG): container finished" podID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerID="c3ba7b533ba80826438a342abf033f36a84441b2f163197359b0d5a790376c51" exitCode=0
Oct 30 21:05:50.144303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.144279    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx" event={"ID":"6f2137c2-d968-4045-8caf-bec6fa171bc2","Type":"ContainerDied","Data":"c3ba7b533ba80826438a342abf033f36a84441b2f163197359b0d5a790376c51"}
Oct 30 21:05:50.151317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.151286    4633 generic.go:356] "Generic (PLEG): container finished" podID="813575cb-0644-4a95-9881-caba2b383d98" containerID="2c4af443964ccade9ca54556a37f72e198df8e4ca38dc3e8dc2801886967ad5b" exitCode=0
Oct 30 21:05:50.151425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:50.151330    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k" event={"ID":"813575cb-0644-4a95-9881-caba2b383d98","Type":"ContainerDied","Data":"2c4af443964ccade9ca54556a37f72e198df8e4ca38dc3e8dc2801886967ad5b"}
Oct 30 21:05:51.189636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:51.188539    4633 generic.go:356] "Generic (PLEG): container finished" podID="95af1760-9426-4411-97c4-19674d809ed7" containerID="22631465b60892a5e8b3ca98536c6e9b7b6c7722100771704ac84baf19f2f648" exitCode=0
Oct 30 21:05:51.192213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:51.190118    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v" event={"ID":"95af1760-9426-4411-97c4-19674d809ed7","Type":"ContainerDied","Data":"22631465b60892a5e8b3ca98536c6e9b7b6c7722100771704ac84baf19f2f648"}
Oct 30 21:05:52.226213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:52.226014    4633 generic.go:356] "Generic (PLEG): container finished" podID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerID="a193862c56b2ecae53bcc0ebe28e984b05ec3695d88eb6749d657d43dec2570d" exitCode=0
Oct 30 21:05:52.226213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:05:52.226141    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj" event={"ID":"0d2dec23-ac5c-4a58-8528-71411db5e8c0","Type":"ContainerDied","Data":"a193862c56b2ecae53bcc0ebe28e984b05ec3695d88eb6749d657d43dec2570d"}
Oct 30 21:05:55.053552 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:05:55.053512    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:06:10.492866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:10.491924    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6" event={"ID":"367837ae-fcd7-44d3-a5d7-012d73bec03b","Type":"ContainerStarted","Data":"9c1eaf355672f264477b7789dd6bbabba2e2e5fade383a0a5abbbed4f52a36ea"}
Oct 30 21:06:10.523123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:10.523051    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6" podStartSLOduration=1.653596898 podStartE2EDuration="45.52302806s" totalImagesPullingTime="43.869431162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:10.521029387 +0000 UTC m=+278.870756784" watchObservedRunningTime="2026-10-30 21:06:10.52302806 +0000 UTC m=+278.872755458"
Oct 30 21:06:11.501406 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:11.501370    4633 generic.go:356] "Generic (PLEG): container finished" podID="96bb663c-3225-4595-83e7-82b322868b20" containerID="bd005655efaebd456316145f1ee69f55c29b9ec6378ec6d3a12f55976019269b" exitCode=0
Oct 30 21:06:11.501872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:11.501461    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k" event={"ID":"96bb663c-3225-4595-83e7-82b322868b20","Type":"ContainerDied","Data":"bd005655efaebd456316145f1ee69f55c29b9ec6378ec6d3a12f55976019269b"}
Oct 30 21:06:11.506922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:11.505430    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl" event={"ID":"0f495ab7-b401-41de-9b30-af7a3f5b6bb5","Type":"ContainerStarted","Data":"52f7b559f2281e4e58260fe4f955866e7c1a31b54e58853e954f9cded99a56c2"}
Oct 30 21:06:11.512120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:11.511891    4633 generic.go:356] "Generic (PLEG): container finished" podID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerID="9c1eaf355672f264477b7789dd6bbabba2e2e5fade383a0a5abbbed4f52a36ea" exitCode=0
Oct 30 21:06:11.512120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:11.511967    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6" event={"ID":"367837ae-fcd7-44d3-a5d7-012d73bec03b","Type":"ContainerDied","Data":"9c1eaf355672f264477b7789dd6bbabba2e2e5fade383a0a5abbbed4f52a36ea"}
Oct 30 21:06:11.565621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:11.563838    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl" podStartSLOduration=2.506284532 podStartE2EDuration="41.563816607s" totalImagesPullingTime="39.057532075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:11.563090256 +0000 UTC m=+279.912817685" watchObservedRunningTime="2026-10-30 21:06:11.563816607 +0000 UTC m=+279.913544004"
Oct 30 21:06:12.539432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.539390    4633 generic.go:356] "Generic (PLEG): container finished" podID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerID="52f7b559f2281e4e58260fe4f955866e7c1a31b54e58853e954f9cded99a56c2" exitCode=0
Oct 30 21:06:12.540224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.540192    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl" event={"ID":"0f495ab7-b401-41de-9b30-af7a3f5b6bb5","Type":"ContainerDied","Data":"52f7b559f2281e4e58260fe4f955866e7c1a31b54e58853e954f9cded99a56c2"}
Oct 30 21:06:12.842382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.842347    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:06:12.859230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.858297    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:06:12.901214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.901182    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-bundle\") pod \"367837ae-fcd7-44d3-a5d7-012d73bec03b\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") "
Oct 30 21:06:12.901399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.901240    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367837ae-fcd7-44d3-a5d7-012d73bec03b-kube-api-access-p9rk8\" (UniqueName: \"kubernetes.io/projected/367837ae-fcd7-44d3-a5d7-012d73bec03b-kube-api-access-p9rk8\") pod \"367837ae-fcd7-44d3-a5d7-012d73bec03b\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") "
Oct 30 21:06:12.901399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.901298    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-util\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-util\") pod \"367837ae-fcd7-44d3-a5d7-012d73bec03b\" (UID: \"367837ae-fcd7-44d3-a5d7-012d73bec03b\") "
Oct 30 21:06:12.905190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.905155    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-bundle" pod "367837ae-fcd7-44d3-a5d7-012d73bec03b" (UID: "367837ae-fcd7-44d3-a5d7-012d73bec03b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:12.906059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.906033    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367837ae-fcd7-44d3-a5d7-012d73bec03b-kube-api-access-p9rk8" pod "367837ae-fcd7-44d3-a5d7-012d73bec03b" (UID: "367837ae-fcd7-44d3-a5d7-012d73bec03b"). InnerVolumeSpecName "kube-api-access-p9rk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:12.909649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:12.909617    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-util" pod "367837ae-fcd7-44d3-a5d7-012d73bec03b" (UID: "367837ae-fcd7-44d3-a5d7-012d73bec03b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.003327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.002566    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-util\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-util\") pod \"96bb663c-3225-4595-83e7-82b322868b20\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") "
Oct 30 21:06:13.003327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.002646    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-bundle\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-bundle\") pod \"96bb663c-3225-4595-83e7-82b322868b20\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") "
Oct 30 21:06:13.003327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.002704    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bb663c-3225-4595-83e7-82b322868b20-kube-api-access-jnd7j\" (UniqueName: \"kubernetes.io/projected/96bb663c-3225-4595-83e7-82b322868b20-kube-api-access-jnd7j\") pod \"96bb663c-3225-4595-83e7-82b322868b20\" (UID: \"96bb663c-3225-4595-83e7-82b322868b20\") "
Oct 30 21:06:13.003327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.002932    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.003327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.002951    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9rk8\" (UniqueName: \"kubernetes.io/projected/367837ae-fcd7-44d3-a5d7-012d73bec03b-kube-api-access-p9rk8\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.003327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.002968    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/367837ae-fcd7-44d3-a5d7-012d73bec03b-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.009037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.003848    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-bundle" pod "96bb663c-3225-4595-83e7-82b322868b20" (UID: "96bb663c-3225-4595-83e7-82b322868b20"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.009037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.007074    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bb663c-3225-4595-83e7-82b322868b20-kube-api-access-jnd7j" pod "96bb663c-3225-4595-83e7-82b322868b20" (UID: "96bb663c-3225-4595-83e7-82b322868b20"). InnerVolumeSpecName "kube-api-access-jnd7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.011783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.011750    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-util" pod "96bb663c-3225-4595-83e7-82b322868b20" (UID: "96bb663c-3225-4595-83e7-82b322868b20"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.104452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.104359    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.104452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.104393    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnd7j\" (UniqueName: \"kubernetes.io/projected/96bb663c-3225-4595-83e7-82b322868b20-kube-api-access-jnd7j\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.104452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.104408    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/96bb663c-3225-4595-83e7-82b322868b20-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.557217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.557178    4633 generic.go:356] "Generic (PLEG): container finished" podID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerID="b9c181c5c1e5c0225ccaa9cdf5b80d3e8ce3c582af29690b7862feb0b70473bd" exitCode=0
Oct 30 21:06:13.557607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.557306    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76" event={"ID":"169245ae-3643-4bfa-b8b3-f5b5e5308d51","Type":"ContainerDied","Data":"b9c181c5c1e5c0225ccaa9cdf5b80d3e8ce3c582af29690b7862feb0b70473bd"}
Oct 30 21:06:13.561878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.561852    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k"
Oct 30 21:06:13.562002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.561951    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpl7k" event={"ID":"96bb663c-3225-4595-83e7-82b322868b20","Type":"ContainerDied","Data":"e2374a4e510cbab2856ef33b8e9847f805bf1e4395e3f6d6b36c87880420b06f"}
Oct 30 21:06:13.562067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.562002    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2374a4e510cbab2856ef33b8e9847f805bf1e4395e3f6d6b36c87880420b06f"
Oct 30 21:06:13.567230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.567208    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6"
Oct 30 21:06:13.568956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.568732    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719h8f6" event={"ID":"367837ae-fcd7-44d3-a5d7-012d73bec03b","Type":"ContainerDied","Data":"e85ad4f52589fe61d039613baa681929a62cdd80e4b19cec6a70ae4eefec637f"}
Oct 30 21:06:13.568956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.568791    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e85ad4f52589fe61d039613baa681929a62cdd80e4b19cec6a70ae4eefec637f"
Oct 30 21:06:13.737233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.737207    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:06:13.813242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.813211    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-bundle\") pod \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") "
Oct 30 21:06:13.813438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.813287    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-util\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-util\") pod \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") "
Oct 30 21:06:13.813438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.813381    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-kube-api-access-27rbc\" (UniqueName: \"kubernetes.io/projected/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-kube-api-access-27rbc\") pod \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\" (UID: \"0f495ab7-b401-41de-9b30-af7a3f5b6bb5\") "
Oct 30 21:06:13.813814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.813788    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-bundle" pod "0f495ab7-b401-41de-9b30-af7a3f5b6bb5" (UID: "0f495ab7-b401-41de-9b30-af7a3f5b6bb5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.817514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.817488    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-kube-api-access-27rbc" pod "0f495ab7-b401-41de-9b30-af7a3f5b6bb5" (UID: "0f495ab7-b401-41de-9b30-af7a3f5b6bb5"). InnerVolumeSpecName "kube-api-access-27rbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:13.819982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.819948    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-util" pod "0f495ab7-b401-41de-9b30-af7a3f5b6bb5" (UID: "0f495ab7-b401-41de-9b30-af7a3f5b6bb5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:13.914209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.914167    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.914209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.914202    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:13.914209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:13.914212    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rbc\" (UniqueName: \"kubernetes.io/projected/0f495ab7-b401-41de-9b30-af7a3f5b6bb5-kube-api-access-27rbc\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:14.573797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.573760    4633 generic.go:356] "Generic (PLEG): container finished" podID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerID="37a12978ea17fcc6fc3f2babb959a09e222b0c846774dcb7ec3d4358566ecae4" exitCode=0
Oct 30 21:06:14.574246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.573844    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx" event={"ID":"6f2137c2-d968-4045-8caf-bec6fa171bc2","Type":"ContainerDied","Data":"37a12978ea17fcc6fc3f2babb959a09e222b0c846774dcb7ec3d4358566ecae4"}
Oct 30 21:06:14.576147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.576123    4633 generic.go:356] "Generic (PLEG): container finished" podID="813575cb-0644-4a95-9881-caba2b383d98" containerID="8a25c336c800fb49f2c8371c850147cc4a5c30bff90e13c0897fee2c12c80719" exitCode=0
Oct 30 21:06:14.576284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.576188    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k" event={"ID":"813575cb-0644-4a95-9881-caba2b383d98","Type":"ContainerDied","Data":"8a25c336c800fb49f2c8371c850147cc4a5c30bff90e13c0897fee2c12c80719"}
Oct 30 21:06:14.578493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.578470    4633 generic.go:356] "Generic (PLEG): container finished" podID="b369efab-35eb-48aa-9f71-9900ee02f727" containerID="decf7f52c9aea25cb071433485d701b8c523ce83d5d839e366fd2852ea6a9a9d" exitCode=0
Oct 30 21:06:14.578621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.578552    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc" event={"ID":"b369efab-35eb-48aa-9f71-9900ee02f727","Type":"ContainerDied","Data":"decf7f52c9aea25cb071433485d701b8c523ce83d5d839e366fd2852ea6a9a9d"}
Oct 30 21:06:14.580774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.580749    4633 generic.go:356] "Generic (PLEG): container finished" podID="2cf875be-7347-490d-9f81-af2730212752" containerID="b959309c412de1c00875df5dc143c5ff1d65fa13d04bae63a0fe361401affcba" exitCode=0
Oct 30 21:06:14.580896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.580838    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9" event={"ID":"2cf875be-7347-490d-9f81-af2730212752","Type":"ContainerDied","Data":"b959309c412de1c00875df5dc143c5ff1d65fa13d04bae63a0fe361401affcba"}
Oct 30 21:06:14.582448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.582429    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:06:14.583102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.583082    4633 generic.go:356] "Generic (PLEG): container finished" podID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerID="5fbf2d6c9466a92b962a6b77a67652bc07adf42c8d2bf2720150ba729350c443" exitCode=1
Oct 30 21:06:14.583184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.583156    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz" event={"ID":"6e527a0b-1fa4-402c-ab56-6dd27aeca974","Type":"ContainerDied","Data":"5fbf2d6c9466a92b962a6b77a67652bc07adf42c8d2bf2720150ba729350c443"}
Oct 30 21:06:14.584934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.584914    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl" event={"ID":"0f495ab7-b401-41de-9b30-af7a3f5b6bb5","Type":"ContainerDied","Data":"673f9f7bf0267714e4ec19ea49c6d9909cfa0bf31db0fc169943bdc8c566d9bf"}
Oct 30 21:06:14.585004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.584946    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c65crl"
Oct 30 21:06:14.585004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.584952    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="673f9f7bf0267714e4ec19ea49c6d9909cfa0bf31db0fc169943bdc8c566d9bf"
Oct 30 21:06:14.716894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.716869    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:06:14.822368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.822335    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-util\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-util\") pod \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") "
Oct 30 21:06:14.822524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.822392    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169245ae-3643-4bfa-b8b3-f5b5e5308d51-kube-api-access-45vv9\" (UniqueName: \"kubernetes.io/projected/169245ae-3643-4bfa-b8b3-f5b5e5308d51-kube-api-access-45vv9\") pod \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") "
Oct 30 21:06:14.822524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.822413    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-bundle\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-bundle\") pod \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\" (UID: \"169245ae-3643-4bfa-b8b3-f5b5e5308d51\") "
Oct 30 21:06:14.822998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.822973    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-bundle" pod "169245ae-3643-4bfa-b8b3-f5b5e5308d51" (UID: "169245ae-3643-4bfa-b8b3-f5b5e5308d51"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:14.824630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.824604    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169245ae-3643-4bfa-b8b3-f5b5e5308d51-kube-api-access-45vv9" pod "169245ae-3643-4bfa-b8b3-f5b5e5308d51" (UID: "169245ae-3643-4bfa-b8b3-f5b5e5308d51"). InnerVolumeSpecName "kube-api-access-45vv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:14.828537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.828506    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-util" pod "169245ae-3643-4bfa-b8b3-f5b5e5308d51" (UID: "169245ae-3643-4bfa-b8b3-f5b5e5308d51"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:14.923126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.923090    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:14.923126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.923125    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45vv9\" (UniqueName: \"kubernetes.io/projected/169245ae-3643-4bfa-b8b3-f5b5e5308d51-kube-api-access-45vv9\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:14.923126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:14.923136    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/169245ae-3643-4bfa-b8b3-f5b5e5308d51-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:15.591298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.591262    4633 generic.go:356] "Generic (PLEG): container finished" podID="95af1760-9426-4411-97c4-19674d809ed7" containerID="e995db60f1632ec93c1851ad9336133e0b4e20dcdb0fba8cc2cafc431dab7ea0" exitCode=0
Oct 30 21:06:15.591880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.591357    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v" event={"ID":"95af1760-9426-4411-97c4-19674d809ed7","Type":"ContainerDied","Data":"e995db60f1632ec93c1851ad9336133e0b4e20dcdb0fba8cc2cafc431dab7ea0"}
Oct 30 21:06:15.593376 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.593355    4633 generic.go:356] "Generic (PLEG): container finished" podID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerID="a1ffc879f39bf3e7317fe1fb88991e7a9396b0381249802e506095bf3aa9979a" exitCode=0
Oct 30 21:06:15.593514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.593435    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj" event={"ID":"0d2dec23-ac5c-4a58-8528-71411db5e8c0","Type":"ContainerDied","Data":"a1ffc879f39bf3e7317fe1fb88991e7a9396b0381249802e506095bf3aa9979a"}
Oct 30 21:06:15.595240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.595215    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76" event={"ID":"169245ae-3643-4bfa-b8b3-f5b5e5308d51","Type":"ContainerDied","Data":"4c51388418591b6ef2fc8c860d02356bddd59f4460bdbea72c6b17284c5e31bf"}
Oct 30 21:06:15.595363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.595261    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c51388418591b6ef2fc8c860d02356bddd59f4460bdbea72c6b17284c5e31bf"
Oct 30 21:06:15.595363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.595300    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cd6j76"
Oct 30 21:06:15.741240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.741206    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:06:15.811616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.811590    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:06:15.815237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.815210    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:06:15.832353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.832315    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:06:15.836381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.836335    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-bundle\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-bundle\") pod \"813575cb-0644-4a95-9881-caba2b383d98\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") "
Oct 30 21:06:15.836542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.836428    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-util\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-util\") pod \"813575cb-0644-4a95-9881-caba2b383d98\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") "
Oct 30 21:06:15.836542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.836461    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813575cb-0644-4a95-9881-caba2b383d98-kube-api-access-9fldf\" (UniqueName: \"kubernetes.io/projected/813575cb-0644-4a95-9881-caba2b383d98-kube-api-access-9fldf\") pod \"813575cb-0644-4a95-9881-caba2b383d98\" (UID: \"813575cb-0644-4a95-9881-caba2b383d98\") "
Oct 30 21:06:15.836891 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.836863    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-bundle" pod "813575cb-0644-4a95-9881-caba2b383d98" (UID: "813575cb-0644-4a95-9881-caba2b383d98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.839630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.839593    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813575cb-0644-4a95-9881-caba2b383d98-kube-api-access-9fldf" pod "813575cb-0644-4a95-9881-caba2b383d98" (UID: "813575cb-0644-4a95-9881-caba2b383d98"). InnerVolumeSpecName "kube-api-access-9fldf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:15.844354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.844295    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-util" pod "813575cb-0644-4a95-9881-caba2b383d98" (UID: "813575cb-0644-4a95-9881-caba2b383d98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.847602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.847576    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:06:15.851379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.851359    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:06:15.937706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.937619    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf875be-7347-490d-9f81-af2730212752-kube-api-access-f9rdp\" (UniqueName: \"kubernetes.io/projected/2cf875be-7347-490d-9f81-af2730212752-kube-api-access-f9rdp\") pod \"2cf875be-7347-490d-9f81-af2730212752\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") "
Oct 30 21:06:15.938408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.937665    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-util\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-util\") pod \"6f2137c2-d968-4045-8caf-bec6fa171bc2\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") "
Oct 30 21:06:15.938546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.938458    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-bundle\") pod \"b369efab-35eb-48aa-9f71-9900ee02f727\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") "
Oct 30 21:06:15.938546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.938518    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e527a0b-1fa4-402c-ab56-6dd27aeca974-kube-api-access-w6kpc\" (UniqueName: \"kubernetes.io/projected/6e527a0b-1fa4-402c-ab56-6dd27aeca974-kube-api-access-w6kpc\") pod \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") "
Oct 30 21:06:15.938633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.938551    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-bundle\") pod \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") "
Oct 30 21:06:15.938633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.938582    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2137c2-d968-4045-8caf-bec6fa171bc2-kube-api-access-rt5jg\" (UniqueName: \"kubernetes.io/projected/6f2137c2-d968-4045-8caf-bec6fa171bc2-kube-api-access-rt5jg\") pod \"6f2137c2-d968-4045-8caf-bec6fa171bc2\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") "
Oct 30 21:06:15.938633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.938625    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b369efab-35eb-48aa-9f71-9900ee02f727-kube-api-access-w48xb\" (UniqueName: \"kubernetes.io/projected/b369efab-35eb-48aa-9f71-9900ee02f727-kube-api-access-w48xb\") pod \"b369efab-35eb-48aa-9f71-9900ee02f727\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") "
Oct 30 21:06:15.938812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.938655    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-util\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-util\") pod \"2cf875be-7347-490d-9f81-af2730212752\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") "
Oct 30 21:06:15.939128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.939091    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-bundle" pod "b369efab-35eb-48aa-9f71-9900ee02f727" (UID: "b369efab-35eb-48aa-9f71-9900ee02f727"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.939244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.939126    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-bundle\") pod \"6f2137c2-d968-4045-8caf-bec6fa171bc2\" (UID: \"6f2137c2-d968-4045-8caf-bec6fa171bc2\") "
Oct 30 21:06:15.939244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.939184    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-util\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-util\") pod \"b369efab-35eb-48aa-9f71-9900ee02f727\" (UID: \"b369efab-35eb-48aa-9f71-9900ee02f727\") "
Oct 30 21:06:15.939244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.939220    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-bundle\") pod \"2cf875be-7347-490d-9f81-af2730212752\" (UID: \"2cf875be-7347-490d-9f81-af2730212752\") "
Oct 30 21:06:15.939411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.939300    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-util\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-util\") pod \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\" (UID: \"6e527a0b-1fa4-402c-ab56-6dd27aeca974\") "
Oct 30 21:06:15.939528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.939504    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-bundle" pod "6e527a0b-1fa4-402c-ab56-6dd27aeca974" (UID: "6e527a0b-1fa4-402c-ab56-6dd27aeca974"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.939683 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.939562    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:15.939792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.939702    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:15.940213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.940194    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fldf\" (UniqueName: \"kubernetes.io/projected/813575cb-0644-4a95-9881-caba2b383d98-kube-api-access-9fldf\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:15.940299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.940217    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/813575cb-0644-4a95-9881-caba2b383d98-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:15.940534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.940512    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf875be-7347-490d-9f81-af2730212752-kube-api-access-f9rdp" pod "2cf875be-7347-490d-9f81-af2730212752" (UID: "2cf875be-7347-490d-9f81-af2730212752"). InnerVolumeSpecName "kube-api-access-f9rdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:15.941470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.941442    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-bundle" pod "2cf875be-7347-490d-9f81-af2730212752" (UID: "2cf875be-7347-490d-9f81-af2730212752"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.941752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.941728    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-bundle" pod "6f2137c2-d968-4045-8caf-bec6fa171bc2" (UID: "6f2137c2-d968-4045-8caf-bec6fa171bc2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.942555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.942508    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b369efab-35eb-48aa-9f71-9900ee02f727-kube-api-access-w48xb" pod "b369efab-35eb-48aa-9f71-9900ee02f727" (UID: "b369efab-35eb-48aa-9f71-9900ee02f727"). InnerVolumeSpecName "kube-api-access-w48xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:15.942865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.942836    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e527a0b-1fa4-402c-ab56-6dd27aeca974-kube-api-access-w6kpc" pod "6e527a0b-1fa4-402c-ab56-6dd27aeca974" (UID: "6e527a0b-1fa4-402c-ab56-6dd27aeca974"). InnerVolumeSpecName "kube-api-access-w6kpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:15.942981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.942964    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2137c2-d968-4045-8caf-bec6fa171bc2-kube-api-access-rt5jg" pod "6f2137c2-d968-4045-8caf-bec6fa171bc2" (UID: "6f2137c2-d968-4045-8caf-bec6fa171bc2"). InnerVolumeSpecName "kube-api-access-rt5jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:15.947263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.947234    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-util" pod "6f2137c2-d968-4045-8caf-bec6fa171bc2" (UID: "6f2137c2-d968-4045-8caf-bec6fa171bc2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.947992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.947973    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-util" pod "2cf875be-7347-490d-9f81-af2730212752" (UID: "2cf875be-7347-490d-9f81-af2730212752"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.948581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.948560    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-util" pod "b369efab-35eb-48aa-9f71-9900ee02f727" (UID: "b369efab-35eb-48aa-9f71-9900ee02f727"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:15.950629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:15.950606    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-util" pod "6e527a0b-1fa4-402c-ab56-6dd27aeca974" (UID: "6e527a0b-1fa4-402c-ab56-6dd27aeca974"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:16.041533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041494    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b369efab-35eb-48aa-9f71-9900ee02f727-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041526    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041536    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041545    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9rdp\" (UniqueName: \"kubernetes.io/projected/2cf875be-7347-490d-9f81-af2730212752-kube-api-access-f9rdp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041555    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041564    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6kpc\" (UniqueName: \"kubernetes.io/projected/6e527a0b-1fa4-402c-ab56-6dd27aeca974-kube-api-access-w6kpc\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041572    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e527a0b-1fa4-402c-ab56-6dd27aeca974-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041581    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt5jg\" (UniqueName: \"kubernetes.io/projected/6f2137c2-d968-4045-8caf-bec6fa171bc2-kube-api-access-rt5jg\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041589    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48xb\" (UniqueName: \"kubernetes.io/projected/b369efab-35eb-48aa-9f71-9900ee02f727-kube-api-access-w48xb\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041596    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cf875be-7347-490d-9f81-af2730212752-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.041849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.041604    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f2137c2-d968-4045-8caf-bec6fa171bc2-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.594582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594544    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"]
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594871    4633 cpu_manager.go:414] "containerMap: removing container" podUID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerName="pull"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594885    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerName="pull"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594894    4633 cpu_manager.go:414] "containerMap: removing container" podUID="813575cb-0644-4a95-9881-caba2b383d98" containerName="pull"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594899    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813575cb-0644-4a95-9881-caba2b383d98" containerName="pull"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594907    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerName="util"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594912    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerName="util"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594921    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594925    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594934    4633 cpu_manager.go:414] "containerMap: removing container" podUID="96bb663c-3225-4595-83e7-82b322868b20" containerName="util"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594939    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bb663c-3225-4595-83e7-82b322868b20" containerName="util"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594951    4633 cpu_manager.go:414] "containerMap: removing container" podUID="813575cb-0644-4a95-9881-caba2b383d98" containerName="util"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594957    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813575cb-0644-4a95-9881-caba2b383d98" containerName="util"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594962    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594967    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594972    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b369efab-35eb-48aa-9f71-9900ee02f727" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594976    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b369efab-35eb-48aa-9f71-9900ee02f727" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594983    4633 cpu_manager.go:414] "containerMap: removing container" podUID="96bb663c-3225-4595-83e7-82b322868b20" containerName="pull"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594987    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bb663c-3225-4595-83e7-82b322868b20" containerName="pull"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594992    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerName="pull"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.594996    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerName="pull"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595001    4633 cpu_manager.go:414] "containerMap: removing container" podUID="96bb663c-3225-4595-83e7-82b322868b20" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595005    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bb663c-3225-4595-83e7-82b322868b20" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595011    4633 cpu_manager.go:414] "containerMap: removing container" podUID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerName="extract"
Oct 30 21:06:16.595029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595015    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595030    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595061    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595068    4633 cpu_manager.go:414] "containerMap: removing container" podUID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595073    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595078    4633 cpu_manager.go:414] "containerMap: removing container" podUID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595082    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595087    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b369efab-35eb-48aa-9f71-9900ee02f727" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595091    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b369efab-35eb-48aa-9f71-9900ee02f727" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595098    4633 cpu_manager.go:414] "containerMap: removing container" podUID="813575cb-0644-4a95-9881-caba2b383d98" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595105    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813575cb-0644-4a95-9881-caba2b383d98" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595114    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595120    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595130    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595134    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595140    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2cf875be-7347-490d-9f81-af2730212752" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595143    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf875be-7347-490d-9f81-af2730212752" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595150    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2cf875be-7347-490d-9f81-af2730212752" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595154    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf875be-7347-490d-9f81-af2730212752" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595161    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595165    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595176    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b369efab-35eb-48aa-9f71-9900ee02f727" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595180    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b369efab-35eb-48aa-9f71-9900ee02f727" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595189    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595196    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595202    4633 cpu_manager.go:414] "containerMap: removing container" podUID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595206    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerName="util"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595211    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2cf875be-7347-490d-9f81-af2730212752" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595214    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf875be-7347-490d-9f81-af2730212752" containerName="pull"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595220    4633 cpu_manager.go:414] "containerMap: removing container" podUID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595224    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595284    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e527a0b-1fa4-402c-ab56-6dd27aeca974" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595292    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f495ab7-b401-41de-9b30-af7a3f5b6bb5" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595298    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b369efab-35eb-48aa-9f71-9900ee02f727" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595305    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bb663c-3225-4595-83e7-82b322868b20" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595309    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="169245ae-3643-4bfa-b8b3-f5b5e5308d51" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595315    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="813575cb-0644-4a95-9881-caba2b383d98" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595320    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2137c2-d968-4045-8caf-bec6fa171bc2" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595325    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf875be-7347-490d-9f81-af2730212752" containerName="extract"
Oct 30 21:06:16.595887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.595331    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="367837ae-fcd7-44d3-a5d7-012d73bec03b" containerName="extract"
Oct 30 21:06:16.600095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.600068    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.601497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.601472    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:06:16.602422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.602278    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz"
Oct 30 21:06:16.602422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.602319    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz" event={"ID":"6e527a0b-1fa4-402c-ab56-6dd27aeca974","Type":"ContainerDied","Data":"98bdb3c0a27d10f65ccf07d20a2d17d16a4513b3250fdb33b0992c05278bf420"}
Oct 30 21:06:16.602422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.602362    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98bdb3c0a27d10f65ccf07d20a2d17d16a4513b3250fdb33b0992c05278bf420"
Oct 30 21:06:16.604415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.604390    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx"
Oct 30 21:06:16.604533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.604412    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azrfdx" event={"ID":"6f2137c2-d968-4045-8caf-bec6fa171bc2","Type":"ContainerDied","Data":"505f5decbbd3a631210d12a42dedf41a99bbd54c5a5ddc58320876c1b1a0cb74"}
Oct 30 21:06:16.604533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.604454    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505f5decbbd3a631210d12a42dedf41a99bbd54c5a5ddc58320876c1b1a0cb74"
Oct 30 21:06:16.606544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.606526    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k"
Oct 30 21:06:16.606544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.606531    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64xc8k" event={"ID":"813575cb-0644-4a95-9881-caba2b383d98","Type":"ContainerDied","Data":"edb3f99c0d441fa7493ef57e912cbdf8ae2f5f10fe42ef712cb466c2293aa84e"}
Oct 30 21:06:16.606741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.606572    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edb3f99c0d441fa7493ef57e912cbdf8ae2f5f10fe42ef712cb466c2293aa84e"
Oct 30 21:06:16.608875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.608857    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc"
Oct 30 21:06:16.609732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.609205    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7gz6gc" event={"ID":"b369efab-35eb-48aa-9f71-9900ee02f727","Type":"ContainerDied","Data":"64667d88555ff6ff31d09e0b6e3215d6f3e23f1bdf9e06bed7af6ebf30419e3b"}
Oct 30 21:06:16.609732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.609245    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64667d88555ff6ff31d09e0b6e3215d6f3e23f1bdf9e06bed7af6ebf30419e3b"
Oct 30 21:06:16.614707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.614688    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:06:16.615983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.615964    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:06:16.616081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.616037    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:06:16.616081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.616070    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:06:16.616200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.616104    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wf7xj\""
Oct 30 21:06:16.616200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.616150    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:06:16.620279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.620258    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9" event={"ID":"2cf875be-7347-490d-9f81-af2730212752","Type":"ContainerDied","Data":"eabb602d77c97dbb98d5ceeee558abf0b93a36910d9bc406a6bece8915080298"}
Oct 30 21:06:16.620359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.620301    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eabb602d77c97dbb98d5ceeee558abf0b93a36910d9bc406a6bece8915080298"
Oct 30 21:06:16.620434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.620421    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bj4m9"
Oct 30 21:06:16.633985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.633955    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"]
Oct 30 21:06:16.747781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.747740    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.747971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.747811    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-tmp\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.747971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.747916    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.747971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.747960    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.748107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.747993    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gld49\" (UniqueName: \"kubernetes.io/projected/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-kube-api-access-gld49\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.748107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.748062    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-manifests\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.767408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.767383    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:06:16.794416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.794392    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:06:16.849354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849262    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2dec23-ac5c-4a58-8528-71411db5e8c0-kube-api-access-8dp2z\" (UniqueName: \"kubernetes.io/projected/0d2dec23-ac5c-4a58-8528-71411db5e8c0-kube-api-access-8dp2z\") pod \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") "
Oct 30 21:06:16.849354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849324    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-bundle\") pod \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") "
Oct 30 21:06:16.849551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849415    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-util\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-util\") pod \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\" (UID: \"0d2dec23-ac5c-4a58-8528-71411db5e8c0\") "
Oct 30 21:06:16.849551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849510    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-tmp\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.849651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849551    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.849651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849584    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.849651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849618    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gld49\" (UniqueName: \"kubernetes.io/projected/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-kube-api-access-gld49\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.849651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849647    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-manifests\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.849887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.849766    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.850225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.850196    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-manifests\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.850487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.850297    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-bundle" pod "0d2dec23-ac5c-4a58-8528-71411db5e8c0" (UID: "0d2dec23-ac5c-4a58-8528-71411db5e8c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:16.852302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.852271    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2dec23-ac5c-4a58-8528-71411db5e8c0-kube-api-access-8dp2z" pod "0d2dec23-ac5c-4a58-8528-71411db5e8c0" (UID: "0d2dec23-ac5c-4a58-8528-71411db5e8c0"). InnerVolumeSpecName "kube-api-access-8dp2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:16.852459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.852442    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.852649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.852629    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-tmp\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.852781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.852765    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.853257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.853234    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.856268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.856247    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-util" pod "0d2dec23-ac5c-4a58-8528-71411db5e8c0" (UID: "0d2dec23-ac5c-4a58-8528-71411db5e8c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:16.858238 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.858215    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gld49\" (UniqueName: \"kubernetes.io/projected/97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa-kube-api-access-gld49\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-6jrt2\" (UID: \"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.921094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.921053    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:16.950839 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.950808    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95af1760-9426-4411-97c4-19674d809ed7-kube-api-access-wl6md\" (UniqueName: \"kubernetes.io/projected/95af1760-9426-4411-97c4-19674d809ed7-kube-api-access-wl6md\") pod \"95af1760-9426-4411-97c4-19674d809ed7\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") "
Oct 30 21:06:16.950984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.950847    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-bundle\") pod \"95af1760-9426-4411-97c4-19674d809ed7\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") "
Oct 30 21:06:16.950984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.950923    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-util\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-util\") pod \"95af1760-9426-4411-97c4-19674d809ed7\" (UID: \"95af1760-9426-4411-97c4-19674d809ed7\") "
Oct 30 21:06:16.951126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.951110    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.951207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.951133    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2dec23-ac5c-4a58-8528-71411db5e8c0-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.951207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.951147    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dp2z\" (UniqueName: \"kubernetes.io/projected/0d2dec23-ac5c-4a58-8528-71411db5e8c0-kube-api-access-8dp2z\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:16.951300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.951283    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-bundle" pod "95af1760-9426-4411-97c4-19674d809ed7" (UID: "95af1760-9426-4411-97c4-19674d809ed7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:16.953088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.953065    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95af1760-9426-4411-97c4-19674d809ed7-kube-api-access-wl6md" pod "95af1760-9426-4411-97c4-19674d809ed7" (UID: "95af1760-9426-4411-97c4-19674d809ed7"). InnerVolumeSpecName "kube-api-access-wl6md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:06:16.959314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:16.959281    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-util" pod "95af1760-9426-4411-97c4-19674d809ed7" (UID: "95af1760-9426-4411-97c4-19674d809ed7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:06:17.052169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.052134    4633 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-util\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:17.052169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.052169    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6md\" (UniqueName: \"kubernetes.io/projected/95af1760-9426-4411-97c4-19674d809ed7-kube-api-access-wl6md\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:17.052344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.052181    4633 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95af1760-9426-4411-97c4-19674d809ed7-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:06:17.217701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.217654    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"]
Oct 30 21:06:17.219408 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:17.219379    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d55b2e_4ffb_48fa_bb4e_83d3b12dc6aa.slice/crio-6c0f67e088bb5c24c1aad5c298683124f4805dbac27e2a811ee1351eceb918e4 WatchSource:0}: Error finding container 6c0f67e088bb5c24c1aad5c298683124f4805dbac27e2a811ee1351eceb918e4: Status 404 returned error can't find the container with id 6c0f67e088bb5c24c1aad5c298683124f4805dbac27e2a811ee1351eceb918e4
Oct 30 21:06:17.631244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.631158    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v" event={"ID":"95af1760-9426-4411-97c4-19674d809ed7","Type":"ContainerDied","Data":"259fde41af621b4ed1af180d07dfe4af99d31a3e93828b16acf3ca536b7191dc"}
Oct 30 21:06:17.631244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.631242    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259fde41af621b4ed1af180d07dfe4af99d31a3e93828b16acf3ca536b7191dc"
Oct 30 21:06:17.631696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.631174    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64427m5v"
Oct 30 21:06:17.632597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.632573    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2" event={"ID":"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa","Type":"ContainerStarted","Data":"6c0f67e088bb5c24c1aad5c298683124f4805dbac27e2a811ee1351eceb918e4"}
Oct 30 21:06:17.634499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.634478    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj"
Oct 30 21:06:17.634611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.634497    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgxvzj" event={"ID":"0d2dec23-ac5c-4a58-8528-71411db5e8c0","Type":"ContainerDied","Data":"535f314a4254a0cc0dd3ca6968964b50a195b122b42e481581bdb1e6cb4cad25"}
Oct 30 21:06:17.634611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:17.634535    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="535f314a4254a0cc0dd3ca6968964b50a195b122b42e481581bdb1e6cb4cad25"
Oct 30 21:06:20.651798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:20.651775    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2" event={"ID":"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa","Type":"ContainerStarted","Data":"a7e33eb6340fc2e9ab8fb069de1ef9052d67b55743402b3b796ce243b1365a8e"}
Oct 30 21:06:21.658587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:21.658554    4633 generic.go:356] "Generic (PLEG): container finished" podID="97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa" containerID="a7e33eb6340fc2e9ab8fb069de1ef9052d67b55743402b3b796ce243b1365a8e" exitCode=0
Oct 30 21:06:21.659004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:21.658598    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2" event={"ID":"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa","Type":"ContainerDied","Data":"a7e33eb6340fc2e9ab8fb069de1ef9052d67b55743402b3b796ce243b1365a8e"}
Oct 30 21:06:22.667218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:22.667169    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2" event={"ID":"97d55b2e-4ffb-48fa-bb4e-83d3b12dc6aa","Type":"ContainerStarted","Data":"e15ee76502c0156853113482ec84ea2872ab9efa2f7d119f92b6b26dbc3c145e"}
Oct 30 21:06:22.667825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:22.667802    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:22.717866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:22.717806    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2" podStartSLOduration=2.763372549 podStartE2EDuration="6.717791997s" totalImagesPullingTime="3.482561944s" totalInitContainerRuntime="471.857504ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:22.716101219 +0000 UTC m=+291.065828614" watchObservedRunningTime="2026-10-30 21:06:22.717791997 +0000 UTC m=+291.067519392"
Oct 30 21:06:24.916211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916172    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"]
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916561    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerName="pull"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916578    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerName="pull"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916600    4633 cpu_manager.go:414] "containerMap: removing container" podUID="95af1760-9426-4411-97c4-19674d809ed7" containerName="pull"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916608    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95af1760-9426-4411-97c4-19674d809ed7" containerName="pull"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916622    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerName="extract"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916629    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerName="extract"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916640    4633 cpu_manager.go:414] "containerMap: removing container" podUID="95af1760-9426-4411-97c4-19674d809ed7" containerName="util"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916646    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95af1760-9426-4411-97c4-19674d809ed7" containerName="util"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916659    4633 cpu_manager.go:414] "containerMap: removing container" podUID="95af1760-9426-4411-97c4-19674d809ed7" containerName="extract"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916688    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95af1760-9426-4411-97c4-19674d809ed7" containerName="extract"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916709    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerName="util"
Oct 30 21:06:24.916711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916716    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerName="util"
Oct 30 21:06:24.917257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916796    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2dec23-ac5c-4a58-8528-71411db5e8c0" containerName="extract"
Oct 30 21:06:24.917257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.916809    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="95af1760-9426-4411-97c4-19674d809ed7" containerName="extract"
Oct 30 21:06:24.919948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.919929    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"
Oct 30 21:06:24.935964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.935941    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 21:06:24.936192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.936179    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4sx5c\""
Oct 30 21:06:24.936583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.936569    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:06:24.947472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:24.947448    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"]
Oct 30 21:06:25.012312 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:06:25.012277    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:06:25.021772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.021751    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e99137c1-345b-4c07-90f8-e65100486ab2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf\" (UID: \"e99137c1-345b-4c07-90f8-e65100486ab2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"
Oct 30 21:06:25.021851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.021785    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwl95\" (UniqueName: \"kubernetes.io/projected/e99137c1-345b-4c07-90f8-e65100486ab2-kube-api-access-xwl95\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf\" (UID: \"e99137c1-345b-4c07-90f8-e65100486ab2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"
Oct 30 21:06:25.122994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.122955    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e99137c1-345b-4c07-90f8-e65100486ab2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf\" (UID: \"e99137c1-345b-4c07-90f8-e65100486ab2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"
Oct 30 21:06:25.122994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.122998    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwl95\" (UniqueName: \"kubernetes.io/projected/e99137c1-345b-4c07-90f8-e65100486ab2-kube-api-access-xwl95\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf\" (UID: \"e99137c1-345b-4c07-90f8-e65100486ab2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"
Oct 30 21:06:25.123435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.123410    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e99137c1-345b-4c07-90f8-e65100486ab2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf\" (UID: \"e99137c1-345b-4c07-90f8-e65100486ab2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"
Oct 30 21:06:25.132861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.132833    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwl95\" (UniqueName: \"kubernetes.io/projected/e99137c1-345b-4c07-90f8-e65100486ab2-kube-api-access-xwl95\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf\" (UID: \"e99137c1-345b-4c07-90f8-e65100486ab2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"
Oct 30 21:06:25.231296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.231212    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"
Oct 30 21:06:25.506594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.506505    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf"]
Oct 30 21:06:25.510683 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:25.510635    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99137c1_345b_4c07_90f8_e65100486ab2.slice/crio-3a05bc6c973cd491e29e0b45cef2766d9fc60ddc9b9c3115b70cbc0028bfe2c8 WatchSource:0}: Error finding container 3a05bc6c973cd491e29e0b45cef2766d9fc60ddc9b9c3115b70cbc0028bfe2c8: Status 404 returned error can't find the container with id 3a05bc6c973cd491e29e0b45cef2766d9fc60ddc9b9c3115b70cbc0028bfe2c8
Oct 30 21:06:25.682108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:25.682071    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf" event={"ID":"e99137c1-345b-4c07-90f8-e65100486ab2","Type":"ContainerStarted","Data":"3a05bc6c973cd491e29e0b45cef2766d9fc60ddc9b9c3115b70cbc0028bfe2c8"}
Oct 30 21:06:27.696993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:27.696955    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf" event={"ID":"e99137c1-345b-4c07-90f8-e65100486ab2","Type":"ContainerStarted","Data":"9c577e5287dd94dda47624a57ed9a3c5f07e244f09f423376fbe9271252dffd7"}
Oct 30 21:06:27.722172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:27.722117    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-v9mvf" podStartSLOduration=1.97602996 podStartE2EDuration="3.722097665s" totalImagesPullingTime="1.746067705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:27.720297947 +0000 UTC m=+296.070025347" watchObservedRunningTime="2026-10-30 21:06:27.722097665 +0000 UTC m=+296.071825061"
Oct 30 21:06:31.524910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.524872    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"]
Oct 30 21:06:31.528217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.528199    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"
Oct 30 21:06:31.536922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.536896    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 21:06:31.537909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.537888    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bs8c8\""
Oct 30 21:06:31.537909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.537886    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 21:06:31.539192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.539172    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"]
Oct 30 21:06:31.680116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.680076    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56j9\" (UniqueName: \"kubernetes.io/projected/fd9d24ac-8a09-432e-a93e-42bacd469863-kube-api-access-b56j9\") pod \"cert-manager-cainjector-5c57bbd4fc-dt855\" (UID: \"fd9d24ac-8a09-432e-a93e-42bacd469863\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"
Oct 30 21:06:31.680299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.680133    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd9d24ac-8a09-432e-a93e-42bacd469863-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dt855\" (UID: \"fd9d24ac-8a09-432e-a93e-42bacd469863\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"
Oct 30 21:06:31.781014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.780926    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56j9\" (UniqueName: \"kubernetes.io/projected/fd9d24ac-8a09-432e-a93e-42bacd469863-kube-api-access-b56j9\") pod \"cert-manager-cainjector-5c57bbd4fc-dt855\" (UID: \"fd9d24ac-8a09-432e-a93e-42bacd469863\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"
Oct 30 21:06:31.781014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.780986    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd9d24ac-8a09-432e-a93e-42bacd469863-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dt855\" (UID: \"fd9d24ac-8a09-432e-a93e-42bacd469863\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"
Oct 30 21:06:31.789883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.789856    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 21:06:31.790360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.790335    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd9d24ac-8a09-432e-a93e-42bacd469863-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-dt855\" (UID: \"fd9d24ac-8a09-432e-a93e-42bacd469863\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"
Oct 30 21:06:31.791148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.791120    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:06:31.791384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.791365    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:06:31.800331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.800308    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 21:06:31.811685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.811642    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56j9\" (UniqueName: \"kubernetes.io/projected/fd9d24ac-8a09-432e-a93e-42bacd469863-kube-api-access-b56j9\") pod \"cert-manager-cainjector-5c57bbd4fc-dt855\" (UID: \"fd9d24ac-8a09-432e-a93e-42bacd469863\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"
Oct 30 21:06:31.816867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.816844    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:06:31.817345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.817323    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:06:31.851907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.851872    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bs8c8\""
Oct 30 21:06:31.859812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:31.859791    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"
Oct 30 21:06:32.146216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:32.146192    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855"]
Oct 30 21:06:32.148226 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:32.148195    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd9d24ac_8a09_432e_a93e_42bacd469863.slice/crio-4b046940e43d0e7117fb00bc4819b3eee4357a217250441338f9e7bd5e69854b WatchSource:0}: Error finding container 4b046940e43d0e7117fb00bc4819b3eee4357a217250441338f9e7bd5e69854b: Status 404 returned error can't find the container with id 4b046940e43d0e7117fb00bc4819b3eee4357a217250441338f9e7bd5e69854b
Oct 30 21:06:32.723993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:32.723959    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855" event={"ID":"fd9d24ac-8a09-432e-a93e-42bacd469863","Type":"ContainerStarted","Data":"4b046940e43d0e7117fb00bc4819b3eee4357a217250441338f9e7bd5e69854b"}
Oct 30 21:06:33.674293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:33.674261    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-6jrt2"
Oct 30 21:06:35.740828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:35.740789    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855" event={"ID":"fd9d24ac-8a09-432e-a93e-42bacd469863","Type":"ContainerStarted","Data":"47ba069bac8ff0f0d9807795b1d7d63393407229894db702e34bdae124546246"}
Oct 30 21:06:35.760365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:35.760318    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-dt855" podStartSLOduration=1.978613232 podStartE2EDuration="4.76030395s" totalImagesPullingTime="2.781690718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:35.759711113 +0000 UTC m=+304.109438505" watchObservedRunningTime="2026-10-30 21:06:35.76030395 +0000 UTC m=+304.110031346"
Oct 30 21:06:38.628562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.628524    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"]
Oct 30 21:06:38.652231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.652202    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"]
Oct 30 21:06:38.652391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.652300    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"
Oct 30 21:06:38.661977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.661951    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 21:06:38.661977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.661949    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:06:38.662145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.661995    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:06:38.662145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.662042    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nwqf5\""
Oct 30 21:06:38.734755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.734708    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vt8\" (UniqueName: \"kubernetes.io/projected/3fd3f3dc-8243-47ce-9edd-26c8b7974922-kube-api-access-69vt8\") pod \"dns-operator-controller-manager-8557cbdb84-gbnpd\" (UID: \"3fd3f3dc-8243-47ce-9edd-26c8b7974922\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"
Oct 30 21:06:38.836037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.835999    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vt8\" (UniqueName: \"kubernetes.io/projected/3fd3f3dc-8243-47ce-9edd-26c8b7974922-kube-api-access-69vt8\") pod \"dns-operator-controller-manager-8557cbdb84-gbnpd\" (UID: \"3fd3f3dc-8243-47ce-9edd-26c8b7974922\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"
Oct 30 21:06:38.844225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.844200    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vt8\" (UniqueName: \"kubernetes.io/projected/3fd3f3dc-8243-47ce-9edd-26c8b7974922-kube-api-access-69vt8\") pod \"dns-operator-controller-manager-8557cbdb84-gbnpd\" (UID: \"3fd3f3dc-8243-47ce-9edd-26c8b7974922\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"
Oct 30 21:06:38.978121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:38.978088    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"
Oct 30 21:06:39.256983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:39.256959    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"]
Oct 30 21:06:39.258929 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:39.258908    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd3f3dc_8243_47ce_9edd_26c8b7974922.slice/crio-accfba6b6ec587710e6e85f5993b9035c63a2eb53d9f381798059d392b50b128 WatchSource:0}: Error finding container accfba6b6ec587710e6e85f5993b9035c63a2eb53d9f381798059d392b50b128: Status 404 returned error can't find the container with id accfba6b6ec587710e6e85f5993b9035c63a2eb53d9f381798059d392b50b128
Oct 30 21:06:39.261372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:39.261354    4633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:06:39.761632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:39.761599    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd" event={"ID":"3fd3f3dc-8243-47ce-9edd-26c8b7974922","Type":"ContainerStarted","Data":"accfba6b6ec587710e6e85f5993b9035c63a2eb53d9f381798059d392b50b128"}
Oct 30 21:06:40.750983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.750944    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-drc9p"]
Oct 30 21:06:40.754761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.754734    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-drc9p"
Oct 30 21:06:40.762539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.762229    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-nmzqs\""
Oct 30 21:06:40.765733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.765692    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-drc9p"]
Oct 30 21:06:40.855649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.855605    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx274\" (UniqueName: \"kubernetes.io/projected/8eee4c11-10b4-4e3e-aa0e-5216283d08aa-kube-api-access-kx274\") pod \"cert-manager-597b95cb84-drc9p\" (UID: \"8eee4c11-10b4-4e3e-aa0e-5216283d08aa\") " pod="cert-manager/cert-manager-597b95cb84-drc9p"
Oct 30 21:06:40.855649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.855648    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eee4c11-10b4-4e3e-aa0e-5216283d08aa-bound-sa-token\") pod \"cert-manager-597b95cb84-drc9p\" (UID: \"8eee4c11-10b4-4e3e-aa0e-5216283d08aa\") " pod="cert-manager/cert-manager-597b95cb84-drc9p"
Oct 30 21:06:40.956990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.956950    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx274\" (UniqueName: \"kubernetes.io/projected/8eee4c11-10b4-4e3e-aa0e-5216283d08aa-kube-api-access-kx274\") pod \"cert-manager-597b95cb84-drc9p\" (UID: \"8eee4c11-10b4-4e3e-aa0e-5216283d08aa\") " pod="cert-manager/cert-manager-597b95cb84-drc9p"
Oct 30 21:06:40.957183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.957009    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eee4c11-10b4-4e3e-aa0e-5216283d08aa-bound-sa-token\") pod \"cert-manager-597b95cb84-drc9p\" (UID: \"8eee4c11-10b4-4e3e-aa0e-5216283d08aa\") " pod="cert-manager/cert-manager-597b95cb84-drc9p"
Oct 30 21:06:40.965623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.965595    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx274\" (UniqueName: \"kubernetes.io/projected/8eee4c11-10b4-4e3e-aa0e-5216283d08aa-kube-api-access-kx274\") pod \"cert-manager-597b95cb84-drc9p\" (UID: \"8eee4c11-10b4-4e3e-aa0e-5216283d08aa\") " pod="cert-manager/cert-manager-597b95cb84-drc9p"
Oct 30 21:06:40.968111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:40.968085    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8eee4c11-10b4-4e3e-aa0e-5216283d08aa-bound-sa-token\") pod \"cert-manager-597b95cb84-drc9p\" (UID: \"8eee4c11-10b4-4e3e-aa0e-5216283d08aa\") " pod="cert-manager/cert-manager-597b95cb84-drc9p"
Oct 30 21:06:41.069265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:41.069181    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-drc9p"
Oct 30 21:06:41.356268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:41.356237    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-drc9p"]
Oct 30 21:06:41.357552 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:41.357522    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8eee4c11_10b4_4e3e_aa0e_5216283d08aa.slice/crio-cfb8f8de8499edc420d3cf00178fa98183d919eb987999a93c14a70ac662e528 WatchSource:0}: Error finding container cfb8f8de8499edc420d3cf00178fa98183d919eb987999a93c14a70ac662e528: Status 404 returned error can't find the container with id cfb8f8de8499edc420d3cf00178fa98183d919eb987999a93c14a70ac662e528
Oct 30 21:06:41.776571 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:41.776532    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-drc9p" event={"ID":"8eee4c11-10b4-4e3e-aa0e-5216283d08aa","Type":"ContainerStarted","Data":"058b6d1be880c4b8b9770f264391c944adb293e2760dcd13b7893ded1c3b5275"}
Oct 30 21:06:41.776986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:41.776586    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-drc9p" event={"ID":"8eee4c11-10b4-4e3e-aa0e-5216283d08aa","Type":"ContainerStarted","Data":"cfb8f8de8499edc420d3cf00178fa98183d919eb987999a93c14a70ac662e528"}
Oct 30 21:06:41.795706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:41.795374    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-drc9p" podStartSLOduration=1.795356581 podStartE2EDuration="1.795356581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:41.79338082 +0000 UTC m=+310.143108253" watchObservedRunningTime="2026-10-30 21:06:41.795356581 +0000 UTC m=+310.145083977"
Oct 30 21:06:42.283180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.283104    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"]
Oct 30 21:06:42.286351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.286333    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"
Oct 30 21:06:42.306163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.306126    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"]
Oct 30 21:06:42.306774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.306744    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ph42x\""
Oct 30 21:06:42.372201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.372168    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9fd\" (UniqueName: \"kubernetes.io/projected/eb1ebff5-79ec-4034-aee6-76e8bd0f4514-kube-api-access-pg9fd\") pod \"limitador-operator-controller-manager-5778bb4449-cpxbl\" (UID: \"eb1ebff5-79ec-4034-aee6-76e8bd0f4514\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"
Oct 30 21:06:42.473242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.473201    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9fd\" (UniqueName: \"kubernetes.io/projected/eb1ebff5-79ec-4034-aee6-76e8bd0f4514-kube-api-access-pg9fd\") pod \"limitador-operator-controller-manager-5778bb4449-cpxbl\" (UID: \"eb1ebff5-79ec-4034-aee6-76e8bd0f4514\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"
Oct 30 21:06:42.483331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.483300    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9fd\" (UniqueName: \"kubernetes.io/projected/eb1ebff5-79ec-4034-aee6-76e8bd0f4514-kube-api-access-pg9fd\") pod \"limitador-operator-controller-manager-5778bb4449-cpxbl\" (UID: \"eb1ebff5-79ec-4034-aee6-76e8bd0f4514\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"
Oct 30 21:06:42.598433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.598337    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"
Oct 30 21:06:42.783519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.783477    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd" event={"ID":"3fd3f3dc-8243-47ce-9edd-26c8b7974922","Type":"ContainerStarted","Data":"c5fb1694bb5055d5cf870ef4a8202f4816a9ae1d0d05d2ed2be32559f1a1fc9e"}
Oct 30 21:06:42.784049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.783641    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"
Oct 30 21:06:42.802265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.802219    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd" podStartSLOduration=2.170877556 podStartE2EDuration="4.802205578s" totalImagesPullingTime="2.631328022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:42.801320802 +0000 UTC m=+311.151048224" watchObservedRunningTime="2026-10-30 21:06:42.802205578 +0000 UTC m=+311.151932975"
Oct 30 21:06:42.870498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:42.870472    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"]
Oct 30 21:06:42.872622 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:42.872589    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1ebff5_79ec_4034_aee6_76e8bd0f4514.slice/crio-aff7417a1f177b5abcfd8c16a025b67e8e9457fd871587000a7330e814efe53b WatchSource:0}: Error finding container aff7417a1f177b5abcfd8c16a025b67e8e9457fd871587000a7330e814efe53b: Status 404 returned error can't find the container with id aff7417a1f177b5abcfd8c16a025b67e8e9457fd871587000a7330e814efe53b
Oct 30 21:06:43.790052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:43.790018    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl" event={"ID":"eb1ebff5-79ec-4034-aee6-76e8bd0f4514","Type":"ContainerStarted","Data":"aff7417a1f177b5abcfd8c16a025b67e8e9457fd871587000a7330e814efe53b"}
Oct 30 21:06:45.808496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:45.808457    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl" event={"ID":"eb1ebff5-79ec-4034-aee6-76e8bd0f4514","Type":"ContainerStarted","Data":"d654d006018f9aff464775a36310933dc301e0034e456299a9de0e5ca34c3fa3"}
Oct 30 21:06:45.808981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:45.808614    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"
Oct 30 21:06:45.826723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:45.826653    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl" podStartSLOduration=1.9622991239999998 podStartE2EDuration="3.826639895s" totalImagesPullingTime="1.864340771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:45.824933541 +0000 UTC m=+314.174660938" watchObservedRunningTime="2026-10-30 21:06:45.826639895 +0000 UTC m=+314.176367291"
Oct 30 21:06:50.043122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.043077    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"]
Oct 30 21:06:50.094879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.094843    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"
Oct 30 21:06:50.095060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.094838    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"]
Oct 30 21:06:50.102199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.102178    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j5hgs\""
Oct 30 21:06:50.141337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.141303    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4z9\" (UniqueName: \"kubernetes.io/projected/8a968bc1-6e7b-4391-82bf-00d03fc511fb-kube-api-access-tc4z9\") pod \"authorino-operator-7b7bb4bc77-cnlnx\" (UID: \"8a968bc1-6e7b-4391-82bf-00d03fc511fb\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"
Oct 30 21:06:50.242134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.242092    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4z9\" (UniqueName: \"kubernetes.io/projected/8a968bc1-6e7b-4391-82bf-00d03fc511fb-kube-api-access-tc4z9\") pod \"authorino-operator-7b7bb4bc77-cnlnx\" (UID: \"8a968bc1-6e7b-4391-82bf-00d03fc511fb\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"
Oct 30 21:06:50.260417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.260380    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4z9\" (UniqueName: \"kubernetes.io/projected/8a968bc1-6e7b-4391-82bf-00d03fc511fb-kube-api-access-tc4z9\") pod \"authorino-operator-7b7bb4bc77-cnlnx\" (UID: \"8a968bc1-6e7b-4391-82bf-00d03fc511fb\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"
Oct 30 21:06:50.419477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.419444    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"
Oct 30 21:06:50.696157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.696134    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"]
Oct 30 21:06:50.698329 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:50.698301    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a968bc1_6e7b_4391_82bf_00d03fc511fb.slice/crio-10d937f43d01c84f6c161d0b53b41208c68e783318afe1e151248568bb6cb7f9 WatchSource:0}: Error finding container 10d937f43d01c84f6c161d0b53b41208c68e783318afe1e151248568bb6cb7f9: Status 404 returned error can't find the container with id 10d937f43d01c84f6c161d0b53b41208c68e783318afe1e151248568bb6cb7f9
Oct 30 21:06:50.831204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:50.831171    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx" event={"ID":"8a968bc1-6e7b-4391-82bf-00d03fc511fb","Type":"ContainerStarted","Data":"10d937f43d01c84f6c161d0b53b41208c68e783318afe1e151248568bb6cb7f9"}
Oct 30 21:06:52.843084 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:52.842978    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx" event={"ID":"8a968bc1-6e7b-4391-82bf-00d03fc511fb","Type":"ContainerStarted","Data":"0e0c473cbb08a840703d2cf27d5ecfd4935cc1cda3f3832489edaebcb5788373"}
Oct 30 21:06:52.843511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:52.843103    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"
Oct 30 21:06:52.861491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:52.861442    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx" podStartSLOduration=1.068731538 podStartE2EDuration="2.861428576s" totalImagesPullingTime="1.792697038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:52.85983679 +0000 UTC m=+321.209564186" watchObservedRunningTime="2026-10-30 21:06:52.861428576 +0000 UTC m=+321.211155972"
Oct 30 21:06:53.792052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:53.792012    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gbnpd"
Oct 30 21:06:54.463522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.463483    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"]
Oct 30 21:06:54.467217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.467196    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.480366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.480342    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-2q9hh\""
Oct 30 21:06:54.483905 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.483884    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"]
Oct 30 21:06:54.580626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.580591    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.580626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.580630    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.580859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.580655    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543e365f-965c-4ae4-bd76-69a422e1dd6f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.580859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.580703    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwnn\" (UniqueName: \"kubernetes.io/projected/543e365f-965c-4ae4-bd76-69a422e1dd6f-kube-api-access-xcwnn\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.580859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.580840    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.580951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.580873    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.580951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.580894    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.581014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.580962    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.581046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.581012    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682484    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682552    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682589    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682632    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682723    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682786    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682821    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682853    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543e365f-965c-4ae4-bd76-69a422e1dd6f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.682985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682886    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwnn\" (UniqueName: \"kubernetes.io/projected/543e365f-965c-4ae4-bd76-69a422e1dd6f-kube-api-access-xcwnn\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.683177 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.682992    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.683318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.683251    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.683537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.683481    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.683974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.683947    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.684062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.683960    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543e365f-965c-4ae4-bd76-69a422e1dd6f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.685350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.685321    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.686102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.686079    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.691529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.691499    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwnn\" (UniqueName: \"kubernetes.io/projected/543e365f-965c-4ae4-bd76-69a422e1dd6f-kube-api-access-xcwnn\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.692000 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.691975    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543e365f-965c-4ae4-bd76-69a422e1dd6f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt\" (UID: \"543e365f-965c-4ae4-bd76-69a422e1dd6f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:54.779991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:54.779901    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:55.013160 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:06:55.013116    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:06:55.062946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:55.062918    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"]
Oct 30 21:06:55.065116 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:55.065090    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543e365f_965c_4ae4_bd76_69a422e1dd6f.slice/crio-b2b039bf8b8b27b30330c6271f7308960e8467556f1aeecaa2d41b2029cbc716 WatchSource:0}: Error finding container b2b039bf8b8b27b30330c6271f7308960e8467556f1aeecaa2d41b2029cbc716: Status 404 returned error can't find the container with id b2b039bf8b8b27b30330c6271f7308960e8467556f1aeecaa2d41b2029cbc716
Oct 30 21:06:55.067686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:55.067641    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:06:55.067776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:55.067723    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:06:55.067776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:55.067762    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:06:55.863145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:55.863102    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt" event={"ID":"543e365f-965c-4ae4-bd76-69a422e1dd6f","Type":"ContainerStarted","Data":"f5c6c2ada3015433f6147368f36d23e11157c58dd5ae45ef8cc31ac5c8b2c0b0"}
Oct 30 21:06:55.863594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:55.863202    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt" event={"ID":"543e365f-965c-4ae4-bd76-69a422e1dd6f","Type":"ContainerStarted","Data":"b2b039bf8b8b27b30330c6271f7308960e8467556f1aeecaa2d41b2029cbc716"}
Oct 30 21:06:55.888220 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:55.888164    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt" podStartSLOduration=1.888149281 podStartE2EDuration="1.888149281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:55.886466981 +0000 UTC m=+324.236194378" watchObservedRunningTime="2026-10-30 21:06:55.888149281 +0000 UTC m=+324.237876676"
Oct 30 21:06:56.389645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.389610    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"]
Oct 30 21:06:56.393179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.393161    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.404611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.404588    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"]
Oct 30 21:06:56.423638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.423612    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:06:56.424027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.423701    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:06:56.427604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.427585    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 21:06:56.445816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.445794    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-df2b6\""
Oct 30 21:06:56.498337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.498307    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee562c52-75cd-44c7-80f6-d1ea3b46e873-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.498484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.498380    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6c2w\" (UniqueName: \"kubernetes.io/projected/ee562c52-75cd-44c7-80f6-d1ea3b46e873-kube-api-access-h6c2w\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.498562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.498497    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee562c52-75cd-44c7-80f6-d1ea3b46e873-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.498562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.498545    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee562c52-75cd-44c7-80f6-d1ea3b46e873-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.599690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.599637    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6c2w\" (UniqueName: \"kubernetes.io/projected/ee562c52-75cd-44c7-80f6-d1ea3b46e873-kube-api-access-h6c2w\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.599894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.599714    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee562c52-75cd-44c7-80f6-d1ea3b46e873-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.599894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.599760    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee562c52-75cd-44c7-80f6-d1ea3b46e873-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.599894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.599779    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee562c52-75cd-44c7-80f6-d1ea3b46e873-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.600176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.600150    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee562c52-75cd-44c7-80f6-d1ea3b46e873-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.600176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.600171    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee562c52-75cd-44c7-80f6-d1ea3b46e873-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.600603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.600583    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee562c52-75cd-44c7-80f6-d1ea3b46e873-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.607542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.607512    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6c2w\" (UniqueName: \"kubernetes.io/projected/ee562c52-75cd-44c7-80f6-d1ea3b46e873-kube-api-access-h6c2w\") pod \"kserve-module-controller-manager-7566cf5fc8-4skfk\" (UID: \"ee562c52-75cd-44c7-80f6-d1ea3b46e873\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.707850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.707819    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:56.784424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.784311    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:56.796780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.796753    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:56.814935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.814912    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cpxbl"
Oct 30 21:06:56.868603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.868578    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:56.869795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.869774    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt"
Oct 30 21:06:56.988207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:56.988132    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"]
Oct 30 21:06:56.991796 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:06:56.991761    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee562c52_75cd_44c7_80f6_d1ea3b46e873.slice/crio-ebf024fc0ce7e236c8b9f01f3e128f26e379a7f042e65356eb02178ec3cff8db WatchSource:0}: Error finding container ebf024fc0ce7e236c8b9f01f3e128f26e379a7f042e65356eb02178ec3cff8db: Status 404 returned error can't find the container with id ebf024fc0ce7e236c8b9f01f3e128f26e379a7f042e65356eb02178ec3cff8db
Oct 30 21:06:57.878020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:57.877978    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk" event={"ID":"ee562c52-75cd-44c7-80f6-d1ea3b46e873","Type":"ContainerStarted","Data":"ebf024fc0ce7e236c8b9f01f3e128f26e379a7f042e65356eb02178ec3cff8db"}
Oct 30 21:06:59.891155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:59.891122    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk" event={"ID":"ee562c52-75cd-44c7-80f6-d1ea3b46e873","Type":"ContainerStarted","Data":"307c8d6babb3bd57eed51cf4ddc6e8d49d7bf99d84f15f896ef923b4a2ddad83"}
Oct 30 21:06:59.891561 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:59.891190    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:06:59.911398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:06:59.911349    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk" podStartSLOduration=1.635662036 podStartE2EDuration="3.911336011s" totalImagesPullingTime="2.275673975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:59.910481398 +0000 UTC m=+328.260208810" watchObservedRunningTime="2026-10-30 21:06:59.911336011 +0000 UTC m=+328.261063408"
Oct 30 21:07:03.853297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:03.853265    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cnlnx"
Oct 30 21:07:12.472418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.472386    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:07:12.475853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.475835    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:12.483720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.483663    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:07:12.484524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.484507    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vp64d\""
Oct 30 21:07:12.503470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.503450    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 21:07:12.510740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.510716    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:07:12.650595 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.650564    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjln\" (UniqueName: \"kubernetes.io/projected/75561d01-e2cb-49bf-9bec-69f1b4441b5d-kube-api-access-2wjln\") pod \"limitador-limitador-7c9b87d574-qvg44\" (UID: \"75561d01-e2cb-49bf-9bec-69f1b4441b5d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:12.650882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.650619    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75561d01-e2cb-49bf-9bec-69f1b4441b5d-config-file\") pod \"limitador-limitador-7c9b87d574-qvg44\" (UID: \"75561d01-e2cb-49bf-9bec-69f1b4441b5d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:12.752096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.752013    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75561d01-e2cb-49bf-9bec-69f1b4441b5d-config-file\") pod \"limitador-limitador-7c9b87d574-qvg44\" (UID: \"75561d01-e2cb-49bf-9bec-69f1b4441b5d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:12.752242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.752108    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjln\" (UniqueName: \"kubernetes.io/projected/75561d01-e2cb-49bf-9bec-69f1b4441b5d-kube-api-access-2wjln\") pod \"limitador-limitador-7c9b87d574-qvg44\" (UID: \"75561d01-e2cb-49bf-9bec-69f1b4441b5d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:12.752662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.752641    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75561d01-e2cb-49bf-9bec-69f1b4441b5d-config-file\") pod \"limitador-limitador-7c9b87d574-qvg44\" (UID: \"75561d01-e2cb-49bf-9bec-69f1b4441b5d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:12.760733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.760703    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjln\" (UniqueName: \"kubernetes.io/projected/75561d01-e2cb-49bf-9bec-69f1b4441b5d-kube-api-access-2wjln\") pod \"limitador-limitador-7c9b87d574-qvg44\" (UID: \"75561d01-e2cb-49bf-9bec-69f1b4441b5d\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:12.790321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:12.790293    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:13.073495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.073463    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:07:13.075217 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:07:13.075173    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75561d01_e2cb_49bf_9bec_69f1b4441b5d.slice/crio-99b005a45cfcb49c796f8297613d3157ef471ecca83a0d4afd67d00653f41136 WatchSource:0}: Error finding container 99b005a45cfcb49c796f8297613d3157ef471ecca83a0d4afd67d00653f41136: Status 404 returned error can't find the container with id 99b005a45cfcb49c796f8297613d3157ef471ecca83a0d4afd67d00653f41136
Oct 30 21:07:13.080860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.080833    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7l9n"]
Oct 30 21:07:13.085889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.085871    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n"
Oct 30 21:07:13.091188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.091164    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7l9n"]
Oct 30 21:07:13.100314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.100275    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zlhck\""
Oct 30 21:07:13.256429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.256391    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxg8\" (UniqueName: \"kubernetes.io/projected/6ff858b7-91eb-4dae-8ba1-5a143f5ae215-kube-api-access-xmxg8\") pod \"authorino-56c6c9ddd5-v7l9n\" (UID: \"6ff858b7-91eb-4dae-8ba1-5a143f5ae215\") " pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n"
Oct 30 21:07:13.357285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.357196    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxg8\" (UniqueName: \"kubernetes.io/projected/6ff858b7-91eb-4dae-8ba1-5a143f5ae215-kube-api-access-xmxg8\") pod \"authorino-56c6c9ddd5-v7l9n\" (UID: \"6ff858b7-91eb-4dae-8ba1-5a143f5ae215\") " pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n"
Oct 30 21:07:13.366010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.365980    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxg8\" (UniqueName: \"kubernetes.io/projected/6ff858b7-91eb-4dae-8ba1-5a143f5ae215-kube-api-access-xmxg8\") pod \"authorino-56c6c9ddd5-v7l9n\" (UID: \"6ff858b7-91eb-4dae-8ba1-5a143f5ae215\") " pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n"
Oct 30 21:07:13.398854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.398823    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n"
Oct 30 21:07:13.676098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.676070    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7l9n"]
Oct 30 21:07:13.678275 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:07:13.678235    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ff858b7_91eb_4dae_8ba1_5a143f5ae215.slice/crio-5fcc0cf0ede06a72151357412bbb28d8d8d8fedf8338c46516fc4c3ddab745eb WatchSource:0}: Error finding container 5fcc0cf0ede06a72151357412bbb28d8d8d8fedf8338c46516fc4c3ddab745eb: Status 404 returned error can't find the container with id 5fcc0cf0ede06a72151357412bbb28d8d8d8fedf8338c46516fc4c3ddab745eb
Oct 30 21:07:13.963119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.963032    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n" event={"ID":"6ff858b7-91eb-4dae-8ba1-5a143f5ae215","Type":"ContainerStarted","Data":"5fcc0cf0ede06a72151357412bbb28d8d8d8fedf8338c46516fc4c3ddab745eb"}
Oct 30 21:07:13.964567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:13.964538    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44" event={"ID":"75561d01-e2cb-49bf-9bec-69f1b4441b5d","Type":"ContainerStarted","Data":"99b005a45cfcb49c796f8297613d3157ef471ecca83a0d4afd67d00653f41136"}
Oct 30 21:07:14.972733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:14.972695    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44" event={"ID":"75561d01-e2cb-49bf-9bec-69f1b4441b5d","Type":"ContainerStarted","Data":"cebb50ffac3640477c8fc425073b56abab053e568f27c3df2db78d54c7a328ef"}
Oct 30 21:07:14.973212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:14.972849    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:14.994097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:14.994032    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44" podStartSLOduration=1.718802768 podStartE2EDuration="2.99401346s" totalImagesPullingTime="1.275210692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:14.991579122 +0000 UTC m=+343.341306519" watchObservedRunningTime="2026-10-30 21:07:14.99401346 +0000 UTC m=+343.343740859"
Oct 30 21:07:15.979418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:15.979388    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n" event={"ID":"6ff858b7-91eb-4dae-8ba1-5a143f5ae215","Type":"ContainerStarted","Data":"5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd"}
Oct 30 21:07:15.994277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:15.994227    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n" podStartSLOduration=1.065270197 podStartE2EDuration="2.994213158s" totalImagesPullingTime="1.928942961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:15.992231229 +0000 UTC m=+344.341958625" watchObservedRunningTime="2026-10-30 21:07:15.994213158 +0000 UTC m=+344.343940555"
Oct 30 21:07:20.897012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:20.896979    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-4skfk"
Oct 30 21:07:25.012202 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:07:25.012161    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:07:25.980492 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:25.980453    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qvg44"
Oct 30 21:07:49.154631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.154592    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8rbvc"]
Oct 30 21:07:49.162907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.162879    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:49.166023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.165995    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8rbvc"]
Oct 30 21:07:49.192895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.192860    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 21:07:49.193860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.193842    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mvk8q\""
Oct 30 21:07:49.277833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.277793    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed98730c-b2a7-43a4-8416-b17455ae777f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8rbvc\" (UID: \"ed98730c-b2a7-43a4-8416-b17455ae777f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:49.278002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.277873    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2p6\" (UniqueName: \"kubernetes.io/projected/ed98730c-b2a7-43a4-8416-b17455ae777f-kube-api-access-qh2p6\") pod \"model-serving-api-75bd89fdfc-8rbvc\" (UID: \"ed98730c-b2a7-43a4-8416-b17455ae777f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:49.379112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.379077    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2p6\" (UniqueName: \"kubernetes.io/projected/ed98730c-b2a7-43a4-8416-b17455ae777f-kube-api-access-qh2p6\") pod \"model-serving-api-75bd89fdfc-8rbvc\" (UID: \"ed98730c-b2a7-43a4-8416-b17455ae777f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:49.379273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.379134    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed98730c-b2a7-43a4-8416-b17455ae777f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8rbvc\" (UID: \"ed98730c-b2a7-43a4-8416-b17455ae777f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:49.381810 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.381784    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed98730c-b2a7-43a4-8416-b17455ae777f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8rbvc\" (UID: \"ed98730c-b2a7-43a4-8416-b17455ae777f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:49.391061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.391032    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2p6\" (UniqueName: \"kubernetes.io/projected/ed98730c-b2a7-43a4-8416-b17455ae777f-kube-api-access-qh2p6\") pod \"model-serving-api-75bd89fdfc-8rbvc\" (UID: \"ed98730c-b2a7-43a4-8416-b17455ae777f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:49.486231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.486130    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:49.789602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:49.789572    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8rbvc"]
Oct 30 21:07:49.789929 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:07:49.789907    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded98730c_b2a7_43a4_8416_b17455ae777f.slice/crio-f1fa62cd803891cd5a14e41576a12b3e52578d732eb486b2d05449f10ad7e860 WatchSource:0}: Error finding container f1fa62cd803891cd5a14e41576a12b3e52578d732eb486b2d05449f10ad7e860: Status 404 returned error can't find the container with id f1fa62cd803891cd5a14e41576a12b3e52578d732eb486b2d05449f10ad7e860
Oct 30 21:07:50.166576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:50.165216    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc" event={"ID":"ed98730c-b2a7-43a4-8416-b17455ae777f","Type":"ContainerStarted","Data":"f1fa62cd803891cd5a14e41576a12b3e52578d732eb486b2d05449f10ad7e860"}
Oct 30 21:07:52.179404 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:52.179366    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc" event={"ID":"ed98730c-b2a7-43a4-8416-b17455ae777f","Type":"ContainerStarted","Data":"095ba0dc1857356c6bb7df7fa0f428ba3be75b76330259a4fd6b9152ed21a979"}
Oct 30 21:07:52.179860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:52.179505    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:07:52.203178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:07:52.203119    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc" podStartSLOduration=1.474971489 podStartE2EDuration="3.203104301s" totalImagesPullingTime="1.728132812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:52.200888672 +0000 UTC m=+380.550616068" watchObservedRunningTime="2026-10-30 21:07:52.203104301 +0000 UTC m=+380.552831697"
Oct 30 21:07:55.014038 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:07:55.013994    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:08:03.188370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:03.188340    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-8rbvc"
Oct 30 21:08:25.013383 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:08:25.013292    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:08:31.518164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.518117    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"]
Oct 30 21:08:31.524474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.524453    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.532289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.532268    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:08:31.534473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.534451    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 21:08:31.535790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.535767    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rx79k\""
Oct 30 21:08:31.538452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.538432    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"]
Oct 30 21:08:31.539026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.539008    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:31.539125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.539090    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:31.646851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.646811    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-metrics-certs\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.647044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.646881    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-openshift-service-ca\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.647044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.646969    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-maas-api-tls\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.647044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.647011    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht66t\" (UniqueName: \"kubernetes.io/projected/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-kube-api-access-ht66t\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.748049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.748014    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-openshift-service-ca\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.748049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.748051    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-maas-api-tls\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.748247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.748077    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht66t\" (UniqueName: \"kubernetes.io/projected/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-kube-api-access-ht66t\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.748247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.748147    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-metrics-certs\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.750679 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.750635    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 21:08:31.750871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.750853    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:08:31.751090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.751066    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:08:31.759002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.758977    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:08:31.759343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.759322    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-openshift-service-ca\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.761340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.761320    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-metrics-certs\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.761399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.761325    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-maas-api-tls\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.770081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.770022    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht66t\" (UniqueName: \"kubernetes.io/projected/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-kube-api-access-ht66t\") pod \"maas-api-59c9ddfb7b-jhprt\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") " pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:31.842134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.842109    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rx79k\""
Oct 30 21:08:31.849338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:31.849317    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:32.127252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:32.127229    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"]
Oct 30 21:08:32.129408 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:08:32.129380    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7d853f_2837_41fd_bc1f_a937ef1ebaa6.slice/crio-0426123320ad43918e66e35fd68f939fc20cfb3fbf486307d745548a51f5b342 WatchSource:0}: Error finding container 0426123320ad43918e66e35fd68f939fc20cfb3fbf486307d745548a51f5b342: Status 404 returned error can't find the container with id 0426123320ad43918e66e35fd68f939fc20cfb3fbf486307d745548a51f5b342
Oct 30 21:08:32.257778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:32.257734    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:08:32.396431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:32.396339    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt" event={"ID":"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6","Type":"ContainerStarted","Data":"0426123320ad43918e66e35fd68f939fc20cfb3fbf486307d745548a51f5b342"}
Oct 30 21:08:37.453119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:37.453083    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt" event={"ID":"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6","Type":"ContainerStarted","Data":"5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59"}
Oct 30 21:08:37.453763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:37.453212    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:37.471759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:37.471706    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt" podStartSLOduration=1.95619398 podStartE2EDuration="6.471693381s" totalImagesPullingTime="4.515499401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:37.468920676 +0000 UTC m=+425.818648072" watchObservedRunningTime="2026-10-30 21:08:37.471693381 +0000 UTC m=+425.821420774"
Oct 30 21:08:41.476251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:41.476171    4633 generic.go:356] "Generic (PLEG): container finished" podID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerID="ab9a933dd20e65f913f3c659f394f70598a7d75bfed16a6d432db304a8e68ba5" exitCode=0
Oct 30 21:08:41.476862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:41.476248    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" event={"ID":"bfa29907-7f35-4ba2-8303-54079a98cb5c","Type":"ContainerDied","Data":"ab9a933dd20e65f913f3c659f394f70598a7d75bfed16a6d432db304a8e68ba5"}
Oct 30 21:08:41.476862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:41.476591    4633 scope.go:122] "RemoveContainer" containerID="ab9a933dd20e65f913f3c659f394f70598a7d75bfed16a6d432db304a8e68ba5"
Oct 30 21:08:41.477722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:41.477694    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:08:41.477774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:41.477762    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:08:41.477807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:41.477795    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:08:42.483807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:42.483770    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" event={"ID":"bfa29907-7f35-4ba2-8303-54079a98cb5c","Type":"ContainerStarted","Data":"d81fca1b11402d33d55a80fcb27068d1bf1ced3bdaa35c5cf2e5260f4accfc7c"}
Oct 30 21:08:42.633696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:42.631134    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:08:42.633696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:42.631320    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:08:42.633696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:42.631371    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:08:43.022470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:43.022399    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:08:43.465028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:43.464998    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:43.635695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:43.626195    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:08:43.635695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:43.626286    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:08:44.340724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.340687    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7sp7z"]
Oct 30 21:08:44.344076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.344055    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:44.361265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.361235    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7sp7z"]
Oct 30 21:08:44.361911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.361895    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 21:08:44.469214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.469175    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cq7l\" (UniqueName: \"kubernetes.io/projected/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-kube-api-access-4cq7l\") pod \"authorino-6889cbd8c4-7sp7z\" (UID: \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\") " pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:44.469376 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.469251    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-tls-cert\") pod \"authorino-6889cbd8c4-7sp7z\" (UID: \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\") " pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:44.570742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.570702    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cq7l\" (UniqueName: \"kubernetes.io/projected/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-kube-api-access-4cq7l\") pod \"authorino-6889cbd8c4-7sp7z\" (UID: \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\") " pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:44.570955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.570860    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-tls-cert\") pod \"authorino-6889cbd8c4-7sp7z\" (UID: \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\") " pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:44.573516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.573492    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-tls-cert\") pod \"authorino-6889cbd8c4-7sp7z\" (UID: \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\") " pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:44.578369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.578348    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cq7l\" (UniqueName: \"kubernetes.io/projected/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-kube-api-access-4cq7l\") pod \"authorino-6889cbd8c4-7sp7z\" (UID: \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\") " pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:44.582628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.582606    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7sp7z"]
Oct 30 21:08:44.582846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.582835    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:44.641766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.613325    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66d878ffb-gz9h6"]
Oct 30 21:08:44.641766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.623032    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:08:44.641766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.623084    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:08:44.646325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.646160    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66d878ffb-gz9h6"]
Oct 30 21:08:44.646325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.646261    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:08:44.772444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.772406    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4k7\" (UniqueName: \"kubernetes.io/projected/352d23fb-4597-4fd9-bd52-37cc76866fe4-kube-api-access-6j4k7\") pod \"authorino-66d878ffb-gz9h6\" (UID: \"352d23fb-4597-4fd9-bd52-37cc76866fe4\") " pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:08:44.772627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.772545    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/352d23fb-4597-4fd9-bd52-37cc76866fe4-tls-cert\") pod \"authorino-66d878ffb-gz9h6\" (UID: \"352d23fb-4597-4fd9-bd52-37cc76866fe4\") " pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:08:44.874043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.873922    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/352d23fb-4597-4fd9-bd52-37cc76866fe4-tls-cert\") pod \"authorino-66d878ffb-gz9h6\" (UID: \"352d23fb-4597-4fd9-bd52-37cc76866fe4\") " pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:08:44.874043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.873970    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4k7\" (UniqueName: \"kubernetes.io/projected/352d23fb-4597-4fd9-bd52-37cc76866fe4-kube-api-access-6j4k7\") pod \"authorino-66d878ffb-gz9h6\" (UID: \"352d23fb-4597-4fd9-bd52-37cc76866fe4\") " pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:08:44.877020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.876991    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/352d23fb-4597-4fd9-bd52-37cc76866fe4-tls-cert\") pod \"authorino-66d878ffb-gz9h6\" (UID: \"352d23fb-4597-4fd9-bd52-37cc76866fe4\") " pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:08:44.881940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.881913    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4k7\" (UniqueName: \"kubernetes.io/projected/352d23fb-4597-4fd9-bd52-37cc76866fe4-kube-api-access-6j4k7\") pod \"authorino-66d878ffb-gz9h6\" (UID: \"352d23fb-4597-4fd9-bd52-37cc76866fe4\") " pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:08:44.889087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.889065    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7sp7z"]
Oct 30 21:08:44.891748 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:08:44.891717    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b12df93_1906_4d2b_89d0_bd3e9d041ff3.slice/crio-ed5021a53707d624a23789a4059cc25080b293768aab036ce2395be9e965867a WatchSource:0}: Error finding container ed5021a53707d624a23789a4059cc25080b293768aab036ce2395be9e965867a: Status 404 returned error can't find the container with id ed5021a53707d624a23789a4059cc25080b293768aab036ce2395be9e965867a
Oct 30 21:08:44.973444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:44.973403    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:08:45.283290 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:45.283263    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66d878ffb-gz9h6"]
Oct 30 21:08:45.284586 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:08:45.284557    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod352d23fb_4597_4fd9_bd52_37cc76866fe4.slice/crio-6c50b2dac3e75de60259c447ff2eeb79365f0f719b571ab3a4c256e1ffad2009 WatchSource:0}: Error finding container 6c50b2dac3e75de60259c447ff2eeb79365f0f719b571ab3a4c256e1ffad2009: Status 404 returned error can't find the container with id 6c50b2dac3e75de60259c447ff2eeb79365f0f719b571ab3a4c256e1ffad2009
Oct 30 21:08:45.501164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:45.501122    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66d878ffb-gz9h6" event={"ID":"352d23fb-4597-4fd9-bd52-37cc76866fe4","Type":"ContainerStarted","Data":"6c50b2dac3e75de60259c447ff2eeb79365f0f719b571ab3a4c256e1ffad2009"}
Oct 30 21:08:45.502359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:45.502333    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z" event={"ID":"4b12df93-1906-4d2b-89d0-bd3e9d041ff3","Type":"ContainerStarted","Data":"ed5021a53707d624a23789a4059cc25080b293768aab036ce2395be9e965867a"}
Oct 30 21:08:45.637753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:45.633115    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:08:45.637753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:45.633181    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:08:46.543478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.543440    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66d878ffb-gz9h6" event={"ID":"352d23fb-4597-4fd9-bd52-37cc76866fe4","Type":"ContainerStarted","Data":"89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01"}
Oct 30 21:08:46.552725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.551151    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z" event={"ID":"4b12df93-1906-4d2b-89d0-bd3e9d041ff3","Type":"ContainerStarted","Data":"993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821"}
Oct 30 21:08:46.552725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.551255    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z" podUID="4b12df93-1906-4d2b-89d0-bd3e9d041ff3" containerName="authorino" containerID="cri-o://993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821" gracePeriod=30
Oct 30 21:08:46.568501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.567136    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-66d878ffb-gz9h6" podStartSLOduration=1.876506175 podStartE2EDuration="2.567121258s" totalImagesPullingTime="690.615083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:46.565255869 +0000 UTC m=+434.914983267" watchObservedRunningTime="2026-10-30 21:08:46.567121258 +0000 UTC m=+434.916848645"
Oct 30 21:08:46.594437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.594361    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z" podStartSLOduration=1.815967227 podStartE2EDuration="2.594347345s" totalImagesPullingTime="778.380118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:46.592909142 +0000 UTC m=+434.942636540" watchObservedRunningTime="2026-10-30 21:08:46.594347345 +0000 UTC m=+434.944074742"
Oct 30 21:08:46.624499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.624054    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:08:46.624499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.624117    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:08:46.642823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.641466    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7l9n"]
Oct 30 21:08:46.648219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.647795    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n" podUID="6ff858b7-91eb-4dae-8ba1-5a143f5ae215" containerName="authorino" containerID="cri-o://5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd" gracePeriod=30
Oct 30 21:08:46.871713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.870085    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:46.894217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.894182    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-kube-api-access-4cq7l\" (UniqueName: \"kubernetes.io/projected/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-kube-api-access-4cq7l\") pod \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\" (UID: \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\") "
Oct 30 21:08:46.894398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.894225    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-tls-cert\" (UniqueName: \"kubernetes.io/secret/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-tls-cert\") pod \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\" (UID: \"4b12df93-1906-4d2b-89d0-bd3e9d041ff3\") "
Oct 30 21:08:46.900310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.900270    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-kube-api-access-4cq7l" pod "4b12df93-1906-4d2b-89d0-bd3e9d041ff3" (UID: "4b12df93-1906-4d2b-89d0-bd3e9d041ff3"). InnerVolumeSpecName "kube-api-access-4cq7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:46.909546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.909501    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-tls-cert" pod "4b12df93-1906-4d2b-89d0-bd3e9d041ff3" (UID: "4b12df93-1906-4d2b-89d0-bd3e9d041ff3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:46.962472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.962448    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n"
Oct 30 21:08:46.995499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.995460    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff858b7-91eb-4dae-8ba1-5a143f5ae215-kube-api-access-xmxg8\" (UniqueName: \"kubernetes.io/projected/6ff858b7-91eb-4dae-8ba1-5a143f5ae215-kube-api-access-xmxg8\") pod \"6ff858b7-91eb-4dae-8ba1-5a143f5ae215\" (UID: \"6ff858b7-91eb-4dae-8ba1-5a143f5ae215\") "
Oct 30 21:08:46.995792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.995773    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cq7l\" (UniqueName: \"kubernetes.io/projected/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-kube-api-access-4cq7l\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:46.995849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:46.995800    4633 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b12df93-1906-4d2b-89d0-bd3e9d041ff3-tls-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:47.006099 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.002875    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff858b7-91eb-4dae-8ba1-5a143f5ae215-kube-api-access-xmxg8" pod "6ff858b7-91eb-4dae-8ba1-5a143f5ae215" (UID: "6ff858b7-91eb-4dae-8ba1-5a143f5ae215"). InnerVolumeSpecName "kube-api-access-xmxg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:47.096745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.096635    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxg8\" (UniqueName: \"kubernetes.io/projected/6ff858b7-91eb-4dae-8ba1-5a143f5ae215-kube-api-access-xmxg8\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:47.558086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.558042    4633 generic.go:356] "Generic (PLEG): container finished" podID="4b12df93-1906-4d2b-89d0-bd3e9d041ff3" containerID="993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821" exitCode=0
Oct 30 21:08:47.558606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.558167    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z" event={"ID":"4b12df93-1906-4d2b-89d0-bd3e9d041ff3","Type":"ContainerDied","Data":"993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821"}
Oct 30 21:08:47.558606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.558212    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z" event={"ID":"4b12df93-1906-4d2b-89d0-bd3e9d041ff3","Type":"ContainerDied","Data":"ed5021a53707d624a23789a4059cc25080b293768aab036ce2395be9e965867a"}
Oct 30 21:08:47.558606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.558237    4633 scope.go:122] "RemoveContainer" containerID="993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821"
Oct 30 21:08:47.558606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.558314    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7sp7z"
Oct 30 21:08:47.560224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.560039    4633 generic.go:356] "Generic (PLEG): container finished" podID="6ff858b7-91eb-4dae-8ba1-5a143f5ae215" containerID="5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd" exitCode=0
Oct 30 21:08:47.560224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.560099    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n"
Oct 30 21:08:47.560224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.560154    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n" event={"ID":"6ff858b7-91eb-4dae-8ba1-5a143f5ae215","Type":"ContainerDied","Data":"5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd"}
Oct 30 21:08:47.560224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.560195    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-v7l9n" event={"ID":"6ff858b7-91eb-4dae-8ba1-5a143f5ae215","Type":"ContainerDied","Data":"5fcc0cf0ede06a72151357412bbb28d8d8d8fedf8338c46516fc4c3ddab745eb"}
Oct 30 21:08:47.569963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.569891    4633 scope.go:122] "RemoveContainer" containerID="993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821"
Oct 30 21:08:47.570332 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:08:47.570303    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821\": container with ID starting with 993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821 not found: ID does not exist" containerID="993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821"
Oct 30 21:08:47.570459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.570345    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821"} err="failed to get container status \"993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821\": rpc error: code = NotFound desc = could not find container \"993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821\": container with ID starting with 993d15d53c7d5c669a5d93f1145e6c758f2b4e8919e1f3b0577f43a5235ac821 not found: ID does not exist"
Oct 30 21:08:47.570459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.570370    4633 scope.go:122] "RemoveContainer" containerID="5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd"
Oct 30 21:08:47.580391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.580370    4633 scope.go:122] "RemoveContainer" containerID="5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd"
Oct 30 21:08:47.580756 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:08:47.580734    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd\": container with ID starting with 5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd not found: ID does not exist" containerID="5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd"
Oct 30 21:08:47.580836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.580765    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd"} err="failed to get container status \"5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd\": rpc error: code = NotFound desc = could not find container \"5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd\": container with ID starting with 5a32492d075f78cb2890c82cdbcff9d1aa78d1ac7e83bac8e311d397dfac03bd not found: ID does not exist"
Oct 30 21:08:47.587118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.587047    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7l9n"]
Oct 30 21:08:47.589131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.589110    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-v7l9n"]
Oct 30 21:08:47.600906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.598267    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7sp7z"]
Oct 30 21:08:47.600906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.600855    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7sp7z"]
Oct 30 21:08:47.620865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.620821    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Oct 30 21:08:47.621057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.620896    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Oct 30 21:08:47.789221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.789186    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b12df93-1906-4d2b-89d0-bd3e9d041ff3" path="/var/lib/kubelet/pods/4b12df93-1906-4d2b-89d0-bd3e9d041ff3/volumes"
Oct 30 21:08:47.789513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:47.789501    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff858b7-91eb-4dae-8ba1-5a143f5ae215" path="/var/lib/kubelet/pods/6ff858b7-91eb-4dae-8ba1-5a143f5ae215/volumes"
Oct 30 21:08:48.624532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:48.624500    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:08:48.625478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:48.625461    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6"
Oct 30 21:08:52.992071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:52.992036    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"]
Oct 30 21:08:52.992580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:52.992270    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt" podUID="1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" containerName="maas-api" containerID="cri-o://5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59" gracePeriod=30
Oct 30 21:08:53.243211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.243145    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:53.352252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.352218    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-kube-api-access-ht66t\" (UniqueName: \"kubernetes.io/projected/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-kube-api-access-ht66t\") pod \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") "
Oct 30 21:08:53.352442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.352270    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-maas-api-tls\") pod \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") "
Oct 30 21:08:53.352442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.352300    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-metrics-certs\") pod \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") "
Oct 30 21:08:53.352442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.352377    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-openshift-service-ca\") pod \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\" (UID: \"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6\") "
Oct 30 21:08:53.352845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.352813    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-openshift-service-ca" pod "1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" (UID: "1a7d853f-2837-41fd-bc1f-a937ef1ebaa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:08:53.354611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.354588    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-maas-api-tls" pod "1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" (UID: "1a7d853f-2837-41fd-bc1f-a937ef1ebaa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:53.354707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.354604    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-kube-api-access-ht66t" pod "1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" (UID: "1a7d853f-2837-41fd-bc1f-a937ef1ebaa6"). InnerVolumeSpecName "kube-api-access-ht66t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:53.354755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.354702    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-metrics-certs" pod "1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" (UID: "1a7d853f-2837-41fd-bc1f-a937ef1ebaa6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:08:53.453311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.453274    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht66t\" (UniqueName: \"kubernetes.io/projected/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-kube-api-access-ht66t\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:53.453311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.453305    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:53.453311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.453314    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:53.453539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.453323    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:53.600212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.600114    4633 generic.go:356] "Generic (PLEG): container finished" podID="1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" containerID="5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59" exitCode=0
Oct 30 21:08:53.600212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.600157    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt" event={"ID":"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6","Type":"ContainerDied","Data":"5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59"}
Oct 30 21:08:53.600435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.600159    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"
Oct 30 21:08:53.600435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.600242    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt" event={"ID":"1a7d853f-2837-41fd-bc1f-a937ef1ebaa6","Type":"ContainerDied","Data":"0426123320ad43918e66e35fd68f939fc20cfb3fbf486307d745548a51f5b342"}
Oct 30 21:08:53.600435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.600270    4633 scope.go:122] "RemoveContainer" containerID="5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59"
Oct 30 21:08:53.610119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.610098    4633 scope.go:122] "RemoveContainer" containerID="5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59"
Oct 30 21:08:53.610387 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:08:53.610368    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59\": container with ID starting with 5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59 not found: ID does not exist" containerID="5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59"
Oct 30 21:08:53.610443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.610395    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59"} err="failed to get container status \"5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59\": rpc error: code = NotFound desc = could not find container \"5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59\": container with ID starting with 5d3007ea188c94bc09f97144c4be68f11872fb765f026bb6105836fb9f80ad59 not found: ID does not exist"
Oct 30 21:08:53.620988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.620966    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"]
Oct 30 21:08:53.625229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.625208    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59c9ddfb7b-jhprt"]
Oct 30 21:08:53.788988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:08:53.788958    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" path="/var/lib/kubelet/pods/1a7d853f-2837-41fd-bc1f-a937ef1ebaa6/volumes"
Oct 30 21:08:55.013980 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:08:55.013944    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:09:21.354740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.354704    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-hv76h"]
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355060    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6ff858b7-91eb-4dae-8ba1-5a143f5ae215" containerName="authorino"
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355071    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff858b7-91eb-4dae-8ba1-5a143f5ae215" containerName="authorino"
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355089    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" containerName="maas-api"
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355094    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" containerName="maas-api"
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355106    4633 cpu_manager.go:414] "containerMap: removing container" podUID="4b12df93-1906-4d2b-89d0-bd3e9d041ff3" containerName="authorino"
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355110    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b12df93-1906-4d2b-89d0-bd3e9d041ff3" containerName="authorino"
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355162    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7d853f-2837-41fd-bc1f-a937ef1ebaa6" containerName="maas-api"
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355172    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b12df93-1906-4d2b-89d0-bd3e9d041ff3" containerName="authorino"
Oct 30 21:09:21.355232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.355181    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff858b7-91eb-4dae-8ba1-5a143f5ae215" containerName="authorino"
Oct 30 21:09:21.358322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.358297    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:21.369374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.369348    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 21:09:21.370166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.370140    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 21:09:21.370295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.370230    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-v4t5v\""
Oct 30 21:09:21.372215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.372191    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-hv76h"]
Oct 30 21:09:21.404974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.404943    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptp75\" (UniqueName: \"kubernetes.io/projected/c0be76e7-9a78-422c-a9a0-48204d207458-kube-api-access-ptp75\") pod \"ai-gateway-controller-7d85675f98-hv76h\" (UID: \"c0be76e7-9a78-422c-a9a0-48204d207458\") " pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:21.405113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.404981    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0be76e7-9a78-422c-a9a0-48204d207458-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-hv76h\" (UID: \"c0be76e7-9a78-422c-a9a0-48204d207458\") " pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:21.505435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.505399    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptp75\" (UniqueName: \"kubernetes.io/projected/c0be76e7-9a78-422c-a9a0-48204d207458-kube-api-access-ptp75\") pod \"ai-gateway-controller-7d85675f98-hv76h\" (UID: \"c0be76e7-9a78-422c-a9a0-48204d207458\") " pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:21.505435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.505436    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0be76e7-9a78-422c-a9a0-48204d207458-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-hv76h\" (UID: \"c0be76e7-9a78-422c-a9a0-48204d207458\") " pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:21.508090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.508066    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0be76e7-9a78-422c-a9a0-48204d207458-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-hv76h\" (UID: \"c0be76e7-9a78-422c-a9a0-48204d207458\") " pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:21.515924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.515898    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptp75\" (UniqueName: \"kubernetes.io/projected/c0be76e7-9a78-422c-a9a0-48204d207458-kube-api-access-ptp75\") pod \"ai-gateway-controller-7d85675f98-hv76h\" (UID: \"c0be76e7-9a78-422c-a9a0-48204d207458\") " pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:21.671321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.671295    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:21.944014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:21.943984    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-hv76h"]
Oct 30 21:09:22.751642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:22.751604    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h" event={"ID":"c0be76e7-9a78-422c-a9a0-48204d207458","Type":"ContainerStarted","Data":"37fb4af93b04a2182a165cef3e64e87fa511404269cab324136203920abc8b81"}
Oct 30 21:09:23.758577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:23.758536    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h" event={"ID":"c0be76e7-9a78-422c-a9a0-48204d207458","Type":"ContainerStarted","Data":"6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580"}
Oct 30 21:09:23.759046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:23.758640    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:23.776822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:23.776774    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h" podStartSLOduration=1.133581052 podStartE2EDuration="2.776759984s" totalImagesPullingTime="1.643178932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:23.774758171 +0000 UTC m=+472.124485567" watchObservedRunningTime="2026-10-30 21:09:23.776759984 +0000 UTC m=+472.126487379"
Oct 30 21:09:24.342302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.342268    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"]
Oct 30 21:09:24.345774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.345758    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:24.353396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.353370    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 21:09:24.357192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.357172    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"]
Oct 30 21:09:24.361588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.361570    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"]
Oct 30 21:09:24.365104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.365089    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:24.377118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.377094    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-54j58\""
Oct 30 21:09:24.380772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.380747    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"]
Oct 30 21:09:24.428050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.428019    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f5c570-a948-4d1c-b92a-de57a980d11b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-q8b56\" (UID: \"f1f5c570-a948-4d1c-b92a-de57a980d11b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:24.428194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.428064    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3af9a3e-eac9-4673-b9da-fc0dc10bab32-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-nb8q8\" (UID: \"d3af9a3e-eac9-4673-b9da-fc0dc10bab32\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:24.428194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.428088    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvs2k\" (UniqueName: \"kubernetes.io/projected/d3af9a3e-eac9-4673-b9da-fc0dc10bab32-kube-api-access-qvs2k\") pod \"payload-processing-55fbbbf9dc-nb8q8\" (UID: \"d3af9a3e-eac9-4673-b9da-fc0dc10bab32\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:24.428292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.428227    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqg57\" (UniqueName: \"kubernetes.io/projected/f1f5c570-a948-4d1c-b92a-de57a980d11b-kube-api-access-sqg57\") pod \"payload-pre-processing-7b98bb678c-q8b56\" (UID: \"f1f5c570-a948-4d1c-b92a-de57a980d11b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:24.529419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.529381    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqg57\" (UniqueName: \"kubernetes.io/projected/f1f5c570-a948-4d1c-b92a-de57a980d11b-kube-api-access-sqg57\") pod \"payload-pre-processing-7b98bb678c-q8b56\" (UID: \"f1f5c570-a948-4d1c-b92a-de57a980d11b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:24.529619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.529468    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f5c570-a948-4d1c-b92a-de57a980d11b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-q8b56\" (UID: \"f1f5c570-a948-4d1c-b92a-de57a980d11b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:24.529619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.529510    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3af9a3e-eac9-4673-b9da-fc0dc10bab32-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-nb8q8\" (UID: \"d3af9a3e-eac9-4673-b9da-fc0dc10bab32\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:24.529619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.529545    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvs2k\" (UniqueName: \"kubernetes.io/projected/d3af9a3e-eac9-4673-b9da-fc0dc10bab32-kube-api-access-qvs2k\") pod \"payload-processing-55fbbbf9dc-nb8q8\" (UID: \"d3af9a3e-eac9-4673-b9da-fc0dc10bab32\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:24.530301 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.530278    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f5c570-a948-4d1c-b92a-de57a980d11b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-q8b56\" (UID: \"f1f5c570-a948-4d1c-b92a-de57a980d11b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:24.530342 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.530308    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3af9a3e-eac9-4673-b9da-fc0dc10bab32-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-nb8q8\" (UID: \"d3af9a3e-eac9-4673-b9da-fc0dc10bab32\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:24.537507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.537476    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvs2k\" (UniqueName: \"kubernetes.io/projected/d3af9a3e-eac9-4673-b9da-fc0dc10bab32-kube-api-access-qvs2k\") pod \"payload-processing-55fbbbf9dc-nb8q8\" (UID: \"d3af9a3e-eac9-4673-b9da-fc0dc10bab32\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:24.537637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.537477    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqg57\" (UniqueName: \"kubernetes.io/projected/f1f5c570-a948-4d1c-b92a-de57a980d11b-kube-api-access-sqg57\") pod \"payload-pre-processing-7b98bb678c-q8b56\" (UID: \"f1f5c570-a948-4d1c-b92a-de57a980d11b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:24.659660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.659621    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:24.681161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.681094    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:24.936442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.936418    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"]
Oct 30 21:09:24.938057 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:09:24.938022    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f5c570_a948_4d1c_b92a_de57a980d11b.slice/crio-2badf2388740803a4003680ff74304ccb701fd7f2d1180bb06e7df0d77aff8c0 WatchSource:0}: Error finding container 2badf2388740803a4003680ff74304ccb701fd7f2d1180bb06e7df0d77aff8c0: Status 404 returned error can't find the container with id 2badf2388740803a4003680ff74304ccb701fd7f2d1180bb06e7df0d77aff8c0
Oct 30 21:09:24.963607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:24.963579    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"]
Oct 30 21:09:24.965392 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:09:24.965362    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3af9a3e_eac9_4673_b9da_fc0dc10bab32.slice/crio-6218069857cac0506e93e0dfd6b96ae7263d98840fefe1f5e43c9d06c84a81c3 WatchSource:0}: Error finding container 6218069857cac0506e93e0dfd6b96ae7263d98840fefe1f5e43c9d06c84a81c3: Status 404 returned error can't find the container with id 6218069857cac0506e93e0dfd6b96ae7263d98840fefe1f5e43c9d06c84a81c3
Oct 30 21:09:25.013754 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:09:25.013718    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:09:25.775238 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:25.775203    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8" event={"ID":"d3af9a3e-eac9-4673-b9da-fc0dc10bab32","Type":"ContainerStarted","Data":"6218069857cac0506e93e0dfd6b96ae7263d98840fefe1f5e43c9d06c84a81c3"}
Oct 30 21:09:25.776735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:25.776646    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56" event={"ID":"f1f5c570-a948-4d1c-b92a-de57a980d11b","Type":"ContainerStarted","Data":"2badf2388740803a4003680ff74304ccb701fd7f2d1180bb06e7df0d77aff8c0"}
Oct 30 21:09:27.792344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:27.792305    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56" event={"ID":"f1f5c570-a948-4d1c-b92a-de57a980d11b","Type":"ContainerStarted","Data":"02007ac2e7c2547193025cbc18e35640de0bf0501f6911e854c192c603d2ae24"}
Oct 30 21:09:27.792827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:27.792370    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:27.793858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:27.793828    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8" event={"ID":"d3af9a3e-eac9-4673-b9da-fc0dc10bab32","Type":"ContainerStarted","Data":"3cb560f0c92fe4f95b228d889b25d28dd031a6c2915b2df78f00100b41bae041"}
Oct 30 21:09:27.793984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:27.793888    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:27.808718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:27.808650    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56" podStartSLOduration=1.614522589 podStartE2EDuration="3.808635405s" totalImagesPullingTime="2.194112816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:27.808493076 +0000 UTC m=+476.158220473" watchObservedRunningTime="2026-10-30 21:09:27.808635405 +0000 UTC m=+476.158362804"
Oct 30 21:09:27.822934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:27.822889    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8" podStartSLOduration=1.659713719 podStartE2EDuration="3.822875534s" totalImagesPullingTime="2.163161815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:27.822215595 +0000 UTC m=+476.171942991" watchObservedRunningTime="2026-10-30 21:09:27.822875534 +0000 UTC m=+476.172602930"
Oct 30 21:09:34.768280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:34.768248    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:35.032818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.032745    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"]
Oct 30 21:09:35.038006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.037989    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:35.044790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.044755    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"]
Oct 30 21:09:35.122478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.122428    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwn8s\" (UniqueName: \"kubernetes.io/projected/325f7a58-6334-4000-ace2-a01551c15805-kube-api-access-fwn8s\") pod \"ai-gateway-controller-7fd78b7fdb-wlnpl\" (UID: \"325f7a58-6334-4000-ace2-a01551c15805\") " pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:35.122732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.122574    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325f7a58-6334-4000-ace2-a01551c15805-metrics-certs\") pod \"ai-gateway-controller-7fd78b7fdb-wlnpl\" (UID: \"325f7a58-6334-4000-ace2-a01551c15805\") " pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:35.223114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.223076    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325f7a58-6334-4000-ace2-a01551c15805-metrics-certs\") pod \"ai-gateway-controller-7fd78b7fdb-wlnpl\" (UID: \"325f7a58-6334-4000-ace2-a01551c15805\") " pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:35.223268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.223149    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwn8s\" (UniqueName: \"kubernetes.io/projected/325f7a58-6334-4000-ace2-a01551c15805-kube-api-access-fwn8s\") pod \"ai-gateway-controller-7fd78b7fdb-wlnpl\" (UID: \"325f7a58-6334-4000-ace2-a01551c15805\") " pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:35.225623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.225598    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325f7a58-6334-4000-ace2-a01551c15805-metrics-certs\") pod \"ai-gateway-controller-7fd78b7fdb-wlnpl\" (UID: \"325f7a58-6334-4000-ace2-a01551c15805\") " pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:35.230823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.230796    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwn8s\" (UniqueName: \"kubernetes.io/projected/325f7a58-6334-4000-ace2-a01551c15805-kube-api-access-fwn8s\") pod \"ai-gateway-controller-7fd78b7fdb-wlnpl\" (UID: \"325f7a58-6334-4000-ace2-a01551c15805\") " pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:35.353613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.353527    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:35.628367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.628346    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"]
Oct 30 21:09:35.631016 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:09:35.630987    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325f7a58_6334_4000_ace2_a01551c15805.slice/crio-fb59099c213b152667c0d38853c9152a0a5e0513e5bc6d28302f14b3fd977cb0 WatchSource:0}: Error finding container fb59099c213b152667c0d38853c9152a0a5e0513e5bc6d28302f14b3fd977cb0: Status 404 returned error can't find the container with id fb59099c213b152667c0d38853c9152a0a5e0513e5bc6d28302f14b3fd977cb0
Oct 30 21:09:35.834617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:35.834585    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl" event={"ID":"325f7a58-6334-4000-ace2-a01551c15805","Type":"ContainerStarted","Data":"fb59099c213b152667c0d38853c9152a0a5e0513e5bc6d28302f14b3fd977cb0"}
Oct 30 21:09:36.841739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:36.841696    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl" event={"ID":"325f7a58-6334-4000-ace2-a01551c15805","Type":"ContainerStarted","Data":"9af2b52f998eaf3dd71fbbdc65a465f24c1427d0a917a3e1529ba0aed004e32d"}
Oct 30 21:09:36.842181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:36.841777    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:36.859421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:36.859375    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl" podStartSLOduration=1.480083757 podStartE2EDuration="1.859363424s" totalImagesPullingTime="379.279667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:36.857083399 +0000 UTC m=+485.206810795" watchObservedRunningTime="2026-10-30 21:09:36.859363424 +0000 UTC m=+485.209090819"
Oct 30 21:09:38.802413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:38.802385    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-q8b56"
Oct 30 21:09:38.802844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:38.802452    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-nb8q8"
Oct 30 21:09:47.849393 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:47.849322    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7fd78b7fdb-wlnpl"
Oct 30 21:09:47.891046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:47.891006    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-hv76h"]
Oct 30 21:09:47.891266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:47.891231    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h" podUID="c0be76e7-9a78-422c-a9a0-48204d207458" containerName="manager" containerID="cri-o://6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580" gracePeriod=30
Oct 30 21:09:48.150074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.150046    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:48.238625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.238589    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0be76e7-9a78-422c-a9a0-48204d207458-kube-api-access-ptp75\" (UniqueName: \"kubernetes.io/projected/c0be76e7-9a78-422c-a9a0-48204d207458-kube-api-access-ptp75\") pod \"c0be76e7-9a78-422c-a9a0-48204d207458\" (UID: \"c0be76e7-9a78-422c-a9a0-48204d207458\") "
Oct 30 21:09:48.238820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.238637    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0be76e7-9a78-422c-a9a0-48204d207458-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0be76e7-9a78-422c-a9a0-48204d207458-metrics-certs\") pod \"c0be76e7-9a78-422c-a9a0-48204d207458\" (UID: \"c0be76e7-9a78-422c-a9a0-48204d207458\") "
Oct 30 21:09:48.241069 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.241039    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0be76e7-9a78-422c-a9a0-48204d207458-metrics-certs" pod "c0be76e7-9a78-422c-a9a0-48204d207458" (UID: "c0be76e7-9a78-422c-a9a0-48204d207458"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:09:48.241235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.241066    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0be76e7-9a78-422c-a9a0-48204d207458-kube-api-access-ptp75" pod "c0be76e7-9a78-422c-a9a0-48204d207458" (UID: "c0be76e7-9a78-422c-a9a0-48204d207458"). InnerVolumeSpecName "kube-api-access-ptp75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:09:48.340232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.340189    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptp75\" (UniqueName: \"kubernetes.io/projected/c0be76e7-9a78-422c-a9a0-48204d207458-kube-api-access-ptp75\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:48.340232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.340225    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0be76e7-9a78-422c-a9a0-48204d207458-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:09:48.910052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.910015    4633 generic.go:356] "Generic (PLEG): container finished" podID="c0be76e7-9a78-422c-a9a0-48204d207458" containerID="6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580" exitCode=0
Oct 30 21:09:48.910052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.910057    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h" event={"ID":"c0be76e7-9a78-422c-a9a0-48204d207458","Type":"ContainerDied","Data":"6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580"}
Oct 30 21:09:48.910573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.910072    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h"
Oct 30 21:09:48.910573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.910107    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-hv76h" event={"ID":"c0be76e7-9a78-422c-a9a0-48204d207458","Type":"ContainerDied","Data":"37fb4af93b04a2182a165cef3e64e87fa511404269cab324136203920abc8b81"}
Oct 30 21:09:48.910573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.910132    4633 scope.go:122] "RemoveContainer" containerID="6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580"
Oct 30 21:09:48.921351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.921326    4633 scope.go:122] "RemoveContainer" containerID="6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580"
Oct 30 21:09:48.921749 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:09:48.921726    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580\": container with ID starting with 6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580 not found: ID does not exist" containerID="6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580"
Oct 30 21:09:48.921875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.921759    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580"} err="failed to get container status \"6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580\": rpc error: code = NotFound desc = could not find container \"6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580\": container with ID starting with 6fd0e6b1bad83871ca83b28bb84e8f11bf92e8fa77d7da99a5079dde6107c580 not found: ID does not exist"
Oct 30 21:09:48.935589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.935561    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-hv76h"]
Oct 30 21:09:48.939741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:48.939720    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-hv76h"]
Oct 30 21:09:49.789047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:49.789001    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0be76e7-9a78-422c-a9a0-48204d207458" path="/var/lib/kubelet/pods/c0be76e7-9a78-422c-a9a0-48204d207458/volumes"
Oct 30 21:09:55.014919 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:09:55.014875    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:09:55.149363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:55.149331    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:09:58.714648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.714614    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"]
Oct 30 21:09:58.715119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.715021    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c0be76e7-9a78-422c-a9a0-48204d207458" containerName="manager"
Oct 30 21:09:58.715119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.715033    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0be76e7-9a78-422c-a9a0-48204d207458" containerName="manager"
Oct 30 21:09:58.715193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.715123    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0be76e7-9a78-422c-a9a0-48204d207458" containerName="manager"
Oct 30 21:09:58.722727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.722703    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.726171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.726143    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:09:58.726340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.726313    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:09:58.727178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.727158    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 21:09:58.727843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.727822    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"]
Oct 30 21:09:58.728897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.728881    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vk6lq\""
Oct 30 21:09:58.843919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.843885    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.844116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.843941    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94d40fd-549b-42c9-82ec-bdacd455c4ab-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.844116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.844018    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.844116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.844045    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.844116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.844092    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.844333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.844223    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdph\" (UniqueName: \"kubernetes.io/projected/d94d40fd-549b-42c9-82ec-bdacd455c4ab-kube-api-access-czdph\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945289    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945327    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdph\" (UniqueName: \"kubernetes.io/projected/d94d40fd-549b-42c9-82ec-bdacd455c4ab-kube-api-access-czdph\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945367    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945397    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94d40fd-549b-42c9-82ec-bdacd455c4ab-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945427    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945458    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945801    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945829    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.945882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.945851    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.947995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.947968    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d94d40fd-549b-42c9-82ec-bdacd455c4ab-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.948267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.948243    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d94d40fd-549b-42c9-82ec-bdacd455c4ab-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:58.954094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:58.954068    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdph\" (UniqueName: \"kubernetes.io/projected/d94d40fd-549b-42c9-82ec-bdacd455c4ab-kube-api-access-czdph\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cx6kv\" (UID: \"d94d40fd-549b-42c9-82ec-bdacd455c4ab\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:59.040942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.040843    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:09:59.305791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.305396    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:09:59.321851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.321817    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"]
Oct 30 21:09:59.325932 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:09:59.325879    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94d40fd_549b_42c9_82ec_bdacd455c4ab.slice/crio-ebf466932afa094d59a08392b56e0aa7c2af00681a149407bc87a2217af4d383 WatchSource:0}: Error finding container ebf466932afa094d59a08392b56e0aa7c2af00681a149407bc87a2217af4d383: Status 404 returned error can't find the container with id ebf466932afa094d59a08392b56e0aa7c2af00681a149407bc87a2217af4d383
Oct 30 21:09:59.525644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.525609    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"]
Oct 30 21:09:59.531194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.531164    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.537960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.537918    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 21:09:59.541978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.541953    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"]
Oct 30 21:09:59.654963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.654924    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.655170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.655007    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8jw\" (UniqueName: \"kubernetes.io/projected/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-kube-api-access-8s8jw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.655170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.655042    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.655313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.655205    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.655313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.655261    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.655313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.655284    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.756799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.756128    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.756799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.756222    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8jw\" (UniqueName: \"kubernetes.io/projected/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-kube-api-access-8s8jw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.756799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.756257    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.756799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.756314    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.756799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.756358    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.756799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.756382    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.756799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.756624    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.757636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.756902    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.757636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.757091    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.760938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.760866    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.761077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.761014    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.776337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.776304    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8jw\" (UniqueName: \"kubernetes.io/projected/eee866b3-a796-49fb-92dc-a4c27f0aa9a6-kube-api-access-8s8jw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qcjfz\" (UID: \"eee866b3-a796-49fb-92dc-a4c27f0aa9a6\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.855128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.854817    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:09:59.979664 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:09:59.979614    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv" event={"ID":"d94d40fd-549b-42c9-82ec-bdacd455c4ab","Type":"ContainerStarted","Data":"ebf466932afa094d59a08392b56e0aa7c2af00681a149407bc87a2217af4d383"}
Oct 30 21:10:00.163911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:00.160337    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"]
Oct 30 21:10:00.164949 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:10:00.164786    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee866b3_a796_49fb_92dc_a4c27f0aa9a6.slice/crio-4608702b6c821c57fcb9fb19fa23925445ab9af102f6723f09e82cd980610b7c WatchSource:0}: Error finding container 4608702b6c821c57fcb9fb19fa23925445ab9af102f6723f09e82cd980610b7c: Status 404 returned error can't find the container with id 4608702b6c821c57fcb9fb19fa23925445ab9af102f6723f09e82cd980610b7c
Oct 30 21:10:00.991228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:00.990816    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz" event={"ID":"eee866b3-a796-49fb-92dc-a4c27f0aa9a6","Type":"ContainerStarted","Data":"4608702b6c821c57fcb9fb19fa23925445ab9af102f6723f09e82cd980610b7c"}
Oct 30 21:10:01.074745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.074700    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"]
Oct 30 21:10:01.080143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.080079    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.085532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.085499    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"]
Oct 30 21:10:01.090750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.089733    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 21:10:01.180146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.179574    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.180146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.179640    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.180146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.179734    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.180146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.179773    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f5a97c-2b36-48ea-ba42-cee113124243-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.180146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.179810    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjqz\" (UniqueName: \"kubernetes.io/projected/50f5a97c-2b36-48ea-ba42-cee113124243-kube-api-access-qpjqz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.180146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.179901    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.281207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.281161    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.281354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.281256    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.281354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.281290    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.281354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.281347    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.281472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.281374    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f5a97c-2b36-48ea-ba42-cee113124243-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.281472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.281410    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjqz\" (UniqueName: \"kubernetes.io/projected/50f5a97c-2b36-48ea-ba42-cee113124243-kube-api-access-qpjqz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.281750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.281727    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.281842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.281786    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.282149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.282100    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.284597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.284569    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50f5a97c-2b36-48ea-ba42-cee113124243-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.284778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.284687    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f5a97c-2b36-48ea-ba42-cee113124243-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.295962 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.294588    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjqz\" (UniqueName: \"kubernetes.io/projected/50f5a97c-2b36-48ea-ba42-cee113124243-kube-api-access-qpjqz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8\" (UID: \"50f5a97c-2b36-48ea-ba42-cee113124243\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.408186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.408140    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:01.481745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.481608    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:10:01.741385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:01.740359    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"]
Oct 30 21:10:01.742744 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:10:01.742716    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f5a97c_2b36_48ea_ba42_cee113124243.slice/crio-4344f69dd5f87fc27ff97e7a003e02e5384e491f381342c134fab8591b916ee6 WatchSource:0}: Error finding container 4344f69dd5f87fc27ff97e7a003e02e5384e491f381342c134fab8591b916ee6: Status 404 returned error can't find the container with id 4344f69dd5f87fc27ff97e7a003e02e5384e491f381342c134fab8591b916ee6
Oct 30 21:10:02.000368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.000262    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8" event={"ID":"50f5a97c-2b36-48ea-ba42-cee113124243","Type":"ContainerStarted","Data":"4344f69dd5f87fc27ff97e7a003e02e5384e491f381342c134fab8591b916ee6"}
Oct 30 21:10:02.001947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.001920    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz" event={"ID":"eee866b3-a796-49fb-92dc-a4c27f0aa9a6","Type":"ContainerStarted","Data":"071d6d0a215b44c81d803a0e8a864ca25bc0c1c66a834d7c9424a21318a1b958"}
Oct 30 21:10:02.003799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.003772    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv" event={"ID":"d94d40fd-549b-42c9-82ec-bdacd455c4ab","Type":"ContainerStarted","Data":"889e854016285009bb9fb8ac0d024f35cd6001fc649a9eee647b6f7796d0bb2a"}
Oct 30 21:10:02.020598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.020541    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz" podStartSLOduration=1.596763711 podStartE2EDuration="3.02052549s" totalImagesPullingTime="1.423761779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:02.018547832 +0000 UTC m=+510.368275264" watchObservedRunningTime="2026-10-30 21:10:02.02052549 +0000 UTC m=+510.370252903"
Oct 30 21:10:02.038284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.038201    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv" podStartSLOduration=2.440967959 podStartE2EDuration="4.038178383s" totalImagesPullingTime="1.597210424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:02.035631831 +0000 UTC m=+510.385359233" watchObservedRunningTime="2026-10-30 21:10:02.038178383 +0000 UTC m=+510.387905783"
Oct 30 21:10:02.041777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.041747    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:10:02.719953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.719914    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lj99j"]
Oct 30 21:10:02.731138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.731099    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.737106 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.736932    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lj99j"]
Oct 30 21:10:02.744657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.744634    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 21:10:02.856150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.856049    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:10:02.896635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.896596    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-home\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.896839 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.896656    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwnq\" (UniqueName: \"kubernetes.io/projected/e1d828e0-1eea-42a3-b932-223813212485-kube-api-access-vkwnq\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.896839 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.896703    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d828e0-1eea-42a3-b932-223813212485-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.896839 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.896814    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.897001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.896880    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.897001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.896931    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.997911    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.997953    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.997974    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-home\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.998162    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwnq\" (UniqueName: \"kubernetes.io/projected/e1d828e0-1eea-42a3-b932-223813212485-kube-api-access-vkwnq\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.998211    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d828e0-1eea-42a3-b932-223813212485-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.998242    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.998401    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.998531    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-home\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:02.998614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:02.998562    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:03.000695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:03.000645    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1d828e0-1eea-42a3-b932-223813212485-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:03.001079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:03.000894    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1d828e0-1eea-42a3-b932-223813212485-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:03.007635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:03.007611    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwnq\" (UniqueName: \"kubernetes.io/projected/e1d828e0-1eea-42a3-b932-223813212485-kube-api-access-vkwnq\") pod \"echo-simulator-kserve-64b6f8f96c-lj99j\" (UID: \"e1d828e0-1eea-42a3-b932-223813212485\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:03.011834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:03.011809    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8" event={"ID":"50f5a97c-2b36-48ea-ba42-cee113124243","Type":"ContainerStarted","Data":"259845cb71dc1e21ef2cd29cd7a84eeeb43bfac78bcf03108bf83070777e7316"}
Oct 30 21:10:03.053986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:03.053940    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:03.349331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:03.349286    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8" podStartSLOduration=2.03910122 podStartE2EDuration="2.3492641s" totalImagesPullingTime="310.16288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:03.036454205 +0000 UTC m=+511.386181601" watchObservedRunningTime="2026-10-30 21:10:03.3492641 +0000 UTC m=+511.698991495"
Oct 30 21:10:03.350022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:03.350005    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lj99j"]
Oct 30 21:10:03.352513 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:10:03.352485    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d828e0_1eea_42a3_b932_223813212485.slice/crio-26707813fcf6cd7e26ec1d8dc4820cd9656a62bdaecd6c1fa9db616b3eb8b676 WatchSource:0}: Error finding container 26707813fcf6cd7e26ec1d8dc4820cd9656a62bdaecd6c1fa9db616b3eb8b676: Status 404 returned error can't find the container with id 26707813fcf6cd7e26ec1d8dc4820cd9656a62bdaecd6c1fa9db616b3eb8b676
Oct 30 21:10:03.409067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:03.409036    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:04.019087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:04.019022    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j" event={"ID":"e1d828e0-1eea-42a3-b932-223813212485","Type":"ContainerStarted","Data":"26707813fcf6cd7e26ec1d8dc4820cd9656a62bdaecd6c1fa9db616b3eb8b676"}
Oct 30 21:10:05.487926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:05.487888    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:10:06.036730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:06.035393    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j" event={"ID":"e1d828e0-1eea-42a3-b932-223813212485","Type":"ContainerStarted","Data":"ccf14bf1e91db3049aab2cf7f07852f6e99a839cb1062b8ce5b8ec32d6e90426"}
Oct 30 21:10:06.053641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:06.053589    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j" podStartSLOduration=1.870903757 podStartE2EDuration="4.053574414s" totalImagesPullingTime="2.182670657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:06.052726553 +0000 UTC m=+514.402453950" watchObservedRunningTime="2026-10-30 21:10:06.053574414 +0000 UTC m=+514.403301809"
Oct 30 21:10:06.054318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:06.054289    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:09.041924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:09.041878    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:10:09.062456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:09.062420    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:10:09.082016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:09.081981    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cx6kv"
Oct 30 21:10:09.121116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:09.121067    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:10:09.855591 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:09.855554    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:10:09.868627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:09.868599    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:10:10.074968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:10.074940    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz"
Oct 30 21:10:11.408778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:11.408739    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:11.426735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:11.426698    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:12.082989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:12.082966    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8"
Oct 30 21:10:13.054264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:13.054226    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:13.066365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:13.066341    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:13.088416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:13.088391    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j"
Oct 30 21:10:22.543573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.543534    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86dd4f6dc-ggqpz"]
Oct 30 21:10:22.547306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.547285    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86dd4f6dc-ggqpz"
Oct 30 21:10:22.561143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.561114    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86dd4f6dc-ggqpz"]
Oct 30 21:10:22.617356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.617315    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgxg\" (UniqueName: \"kubernetes.io/projected/ef9d6d07-61cb-41de-ba16-fdfe4799c877-kube-api-access-szgxg\") pod \"authorino-86dd4f6dc-ggqpz\" (UID: \"ef9d6d07-61cb-41de-ba16-fdfe4799c877\") " pod="kuadrant-system/authorino-86dd4f6dc-ggqpz"
Oct 30 21:10:22.617508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.617385    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef9d6d07-61cb-41de-ba16-fdfe4799c877-tls-cert\") pod \"authorino-86dd4f6dc-ggqpz\" (UID: \"ef9d6d07-61cb-41de-ba16-fdfe4799c877\") " pod="kuadrant-system/authorino-86dd4f6dc-ggqpz"
Oct 30 21:10:22.718598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.718557    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgxg\" (UniqueName: \"kubernetes.io/projected/ef9d6d07-61cb-41de-ba16-fdfe4799c877-kube-api-access-szgxg\") pod \"authorino-86dd4f6dc-ggqpz\" (UID: \"ef9d6d07-61cb-41de-ba16-fdfe4799c877\") " pod="kuadrant-system/authorino-86dd4f6dc-ggqpz"
Oct 30 21:10:22.718785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.718625    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef9d6d07-61cb-41de-ba16-fdfe4799c877-tls-cert\") pod \"authorino-86dd4f6dc-ggqpz\" (UID: \"ef9d6d07-61cb-41de-ba16-fdfe4799c877\") " pod="kuadrant-system/authorino-86dd4f6dc-ggqpz"
Oct 30 21:10:22.721269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.721246    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef9d6d07-61cb-41de-ba16-fdfe4799c877-tls-cert\") pod \"authorino-86dd4f6dc-ggqpz\" (UID: \"ef9d6d07-61cb-41de-ba16-fdfe4799c877\") " pod="kuadrant-system/authorino-86dd4f6dc-ggqpz"
Oct 30 21:10:22.726598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.726568    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgxg\" (UniqueName: \"kubernetes.io/projected/ef9d6d07-61cb-41de-ba16-fdfe4799c877-kube-api-access-szgxg\") pod \"authorino-86dd4f6dc-ggqpz\" (UID: \"ef9d6d07-61cb-41de-ba16-fdfe4799c877\") " pod="kuadrant-system/authorino-86dd4f6dc-ggqpz"
Oct 30 21:10:22.859602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:22.859514    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86dd4f6dc-ggqpz"
Oct 30 21:10:23.135605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:23.135582    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86dd4f6dc-ggqpz"]
Oct 30 21:10:23.137424 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:10:23.137394    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef9d6d07_61cb_41de_ba16_fdfe4799c877.slice/crio-c30b70f8784020f2929ee87e95b6c00ec86983d00202616195f312d8f6078645 WatchSource:0}: Error finding container c30b70f8784020f2929ee87e95b6c00ec86983d00202616195f312d8f6078645: Status 404 returned error can't find the container with id c30b70f8784020f2929ee87e95b6c00ec86983d00202616195f312d8f6078645
Oct 30 21:10:24.145115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:24.145078    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86dd4f6dc-ggqpz" event={"ID":"ef9d6d07-61cb-41de-ba16-fdfe4799c877","Type":"ContainerStarted","Data":"c30b70f8784020f2929ee87e95b6c00ec86983d00202616195f312d8f6078645"}
Oct 30 21:10:25.025616 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:10:25.025570    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:10:25.151713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.151649    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86dd4f6dc-ggqpz" event={"ID":"ef9d6d07-61cb-41de-ba16-fdfe4799c877","Type":"ContainerStarted","Data":"1670dcbfa34d89b5051fb28bd36bdfade746bf1c9ee637b3b07485251346d2f8"}
Oct 30 21:10:25.177966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.177891    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-86dd4f6dc-ggqpz" podStartSLOduration=1.6687958410000001 podStartE2EDuration="3.177871482s" totalImagesPullingTime="1.509075641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:25.175845075 +0000 UTC m=+533.525572473" watchObservedRunningTime="2026-10-30 21:10:25.177871482 +0000 UTC m=+533.527598879"
Oct 30 21:10:25.212133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.212097    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66d878ffb-gz9h6"]
Oct 30 21:10:25.212367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.212337    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-66d878ffb-gz9h6" podUID="352d23fb-4597-4fd9-bd52-37cc76866fe4" containerName="authorino" containerID="cri-o://89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01" gracePeriod=30
Oct 30 21:10:25.473737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.473708    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:10:25.547209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.547177    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352d23fb-4597-4fd9-bd52-37cc76866fe4-kube-api-access-6j4k7\" (UniqueName: \"kubernetes.io/projected/352d23fb-4597-4fd9-bd52-37cc76866fe4-kube-api-access-6j4k7\") pod \"352d23fb-4597-4fd9-bd52-37cc76866fe4\" (UID: \"352d23fb-4597-4fd9-bd52-37cc76866fe4\") "
Oct 30 21:10:25.547388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.547259    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/352d23fb-4597-4fd9-bd52-37cc76866fe4-tls-cert\" (UniqueName: \"kubernetes.io/secret/352d23fb-4597-4fd9-bd52-37cc76866fe4-tls-cert\") pod \"352d23fb-4597-4fd9-bd52-37cc76866fe4\" (UID: \"352d23fb-4597-4fd9-bd52-37cc76866fe4\") "
Oct 30 21:10:25.549471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.549438    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352d23fb-4597-4fd9-bd52-37cc76866fe4-kube-api-access-6j4k7" pod "352d23fb-4597-4fd9-bd52-37cc76866fe4" (UID: "352d23fb-4597-4fd9-bd52-37cc76866fe4"). InnerVolumeSpecName "kube-api-access-6j4k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:10:25.560139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.560103    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352d23fb-4597-4fd9-bd52-37cc76866fe4-tls-cert" pod "352d23fb-4597-4fd9-bd52-37cc76866fe4" (UID: "352d23fb-4597-4fd9-bd52-37cc76866fe4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:10:25.648966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.648883    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j4k7\" (UniqueName: \"kubernetes.io/projected/352d23fb-4597-4fd9-bd52-37cc76866fe4-kube-api-access-6j4k7\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:25.648966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:25.648914    4633 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/352d23fb-4597-4fd9-bd52-37cc76866fe4-tls-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:10:26.159880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.159844    4633 generic.go:356] "Generic (PLEG): container finished" podID="352d23fb-4597-4fd9-bd52-37cc76866fe4" containerID="89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01" exitCode=0
Oct 30 21:10:26.160486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.159907    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66d878ffb-gz9h6"
Oct 30 21:10:26.160486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.159935    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66d878ffb-gz9h6" event={"ID":"352d23fb-4597-4fd9-bd52-37cc76866fe4","Type":"ContainerDied","Data":"89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01"}
Oct 30 21:10:26.160486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.159992    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66d878ffb-gz9h6" event={"ID":"352d23fb-4597-4fd9-bd52-37cc76866fe4","Type":"ContainerDied","Data":"6c50b2dac3e75de60259c447ff2eeb79365f0f719b571ab3a4c256e1ffad2009"}
Oct 30 21:10:26.160486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.160019    4633 scope.go:122] "RemoveContainer" containerID="89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01"
Oct 30 21:10:26.170685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.170645    4633 scope.go:122] "RemoveContainer" containerID="89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01"
Oct 30 21:10:26.171019 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:10:26.170994    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01\": container with ID starting with 89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01 not found: ID does not exist" containerID="89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01"
Oct 30 21:10:26.171087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.171034    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01"} err="failed to get container status \"89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01\": rpc error: code = NotFound desc = could not find container \"89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01\": container with ID starting with 89007c7fd3ae75664471faaea1785289df5a67d0e72432cb1f87219fd7a17b01 not found: ID does not exist"
Oct 30 21:10:26.196063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.196030    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66d878ffb-gz9h6"]
Oct 30 21:10:26.217789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:26.217758    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66d878ffb-gz9h6"]
Oct 30 21:10:27.789524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:27.789488    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352d23fb-4597-4fd9-bd52-37cc76866fe4" path="/var/lib/kubelet/pods/352d23fb-4597-4fd9-bd52-37cc76866fe4/volumes"
Oct 30 21:10:28.743868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.743827    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"]
Oct 30 21:10:28.744622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.744592    4633 cpu_manager.go:414] "containerMap: removing container" podUID="352d23fb-4597-4fd9-bd52-37cc76866fe4" containerName="authorino"
Oct 30 21:10:28.744622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.744616    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352d23fb-4597-4fd9-bd52-37cc76866fe4" containerName="authorino"
Oct 30 21:10:28.744845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.744765    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="352d23fb-4597-4fd9-bd52-37cc76866fe4" containerName="authorino"
Oct 30 21:10:28.750097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.750074    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.763865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.763829    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"]
Oct 30 21:10:28.887574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.887538    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.887574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.887577    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.888126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.887609    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.888126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.887802    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmxn\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-kube-api-access-qvmxn\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.888126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.887860    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.888126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.887928    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.888126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.887980    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.888126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.888012    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.888126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.888058    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.989560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.989525    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.989779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.989572    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.989779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.989593    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.989779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.989745    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.989961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.989831    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmxn\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-kube-api-access-qvmxn\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.989961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.989878    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.989961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.989944    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.990121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.989977    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.990121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.990006    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.990121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.990018    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.990274 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.990116    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.990336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.990297    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.990426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.990380    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.990951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.990891    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.992720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.992700    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.992818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.992760    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.998291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.998237    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmxn\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-kube-api-access-qvmxn\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:28.998834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:28.998817    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-md8nn\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:29.067033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:29.067002    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:29.358023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:29.357998    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"]
Oct 30 21:10:29.360886 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:10:29.360852    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7891f3fc_84c5_4a05_a212_d58c22f26e5a.slice/crio-1ca313f49105733cf6cdcaaf6777cb7741c68787e781bad109ffee8f467b5ece WatchSource:0}: Error finding container 1ca313f49105733cf6cdcaaf6777cb7741c68787e781bad109ffee8f467b5ece: Status 404 returned error can't find the container with id 1ca313f49105733cf6cdcaaf6777cb7741c68787e781bad109ffee8f467b5ece
Oct 30 21:10:29.363268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:29.363234    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:29.363369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:29.363300    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:29.363369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:29.363331    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:30.187618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:30.187571    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" event={"ID":"7891f3fc-84c5-4a05-a212-d58c22f26e5a","Type":"ContainerStarted","Data":"854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53"}
Oct 30 21:10:30.188098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:30.187638    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" event={"ID":"7891f3fc-84c5-4a05-a212-d58c22f26e5a","Type":"ContainerStarted","Data":"1ca313f49105733cf6cdcaaf6777cb7741c68787e781bad109ffee8f467b5ece"}
Oct 30 21:10:30.211237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:30.211182    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podStartSLOduration=2.211164327 podStartE2EDuration="2.211164327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:30.208308465 +0000 UTC m=+538.558035862" watchObservedRunningTime="2026-10-30 21:10:30.211164327 +0000 UTC m=+538.560891724"
Oct 30 21:10:31.067569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:31.067532    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:31.069245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:31.069215    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-md8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 21:10:31.069388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:31.069279    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 21:10:32.068340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:32.068294    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-md8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 21:10:32.068761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:32.068357    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 21:10:33.067413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:33.067380    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-md8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 21:10:33.067624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:33.067432    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 21:10:34.068017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:34.067981    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-md8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 21:10:34.068519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:34.068037    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 21:10:35.067842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:35.067793    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-md8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 21:10:35.068024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:35.067872    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 21:10:36.068313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:36.068277    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-md8nn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 21:10:36.068889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:36.068861    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 21:10:37.673965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:37.673932    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:37.674517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:37.674494    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:37.675175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:37.675146    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:10:43.777881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.777843    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"]
Oct 30 21:10:43.781553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.781531    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.797375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.797330    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"]
Oct 30 21:10:43.852907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.852876    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.853071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.852937    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.853071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.853013    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.853184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.853100    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae247f2-d22b-4593-9935-071384bb7be9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.853184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.853152    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.853184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.853167    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae247f2-d22b-4593-9935-071384bb7be9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.853184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.853181    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae247f2-d22b-4593-9935-071384bb7be9-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.853343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.853231    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdc4\" (UniqueName: \"kubernetes.io/projected/cae247f2-d22b-4593-9935-071384bb7be9-kube-api-access-hxdc4\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.853343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.853293    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.954563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954533    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.954766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954570    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.954766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954588    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae247f2-d22b-4593-9935-071384bb7be9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.954766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954624    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.954766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954639    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae247f2-d22b-4593-9935-071384bb7be9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.954766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954657    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae247f2-d22b-4593-9935-071384bb7be9-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.954766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954702    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdc4\" (UniqueName: \"kubernetes.io/projected/cae247f2-d22b-4593-9935-071384bb7be9-kube-api-access-hxdc4\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.954766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954744    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.955190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.954792    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.955190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.955055    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.955190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.955137    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.955343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.955247    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.955343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.955321    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.955617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.955394    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae247f2-d22b-4593-9935-071384bb7be9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.957240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.957215    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae247f2-d22b-4593-9935-071384bb7be9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.957340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.957264    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae247f2-d22b-4593-9935-071384bb7be9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.962962 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.962938    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae247f2-d22b-4593-9935-071384bb7be9-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:43.963100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:43.963077    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdc4\" (UniqueName: \"kubernetes.io/projected/cae247f2-d22b-4593-9935-071384bb7be9-kube-api-access-hxdc4\") pod \"maas-default-gateway-openshift-default-75bf867b6c-8tv48\" (UID: \"cae247f2-d22b-4593-9935-071384bb7be9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:44.100978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:44.100886    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:44.382075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:44.382050    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"]
Oct 30 21:10:44.383734 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:10:44.383702    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae247f2_d22b_4593_9935_071384bb7be9.slice/crio-3b5445c55551e1210c7a2446aaf80bc023b741744cf64cd8ec4806757f6e5efb WatchSource:0}: Error finding container 3b5445c55551e1210c7a2446aaf80bc023b741744cf64cd8ec4806757f6e5efb: Status 404 returned error can't find the container with id 3b5445c55551e1210c7a2446aaf80bc023b741744cf64cd8ec4806757f6e5efb
Oct 30 21:10:44.385991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:44.385951    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:44.386113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:44.386020    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:44.386113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:44.386051    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:45.274098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:45.274062    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" event={"ID":"cae247f2-d22b-4593-9935-071384bb7be9","Type":"ContainerStarted","Data":"bd66494833a4348387276377bfdf234196b11c222fe0334adb14aab21663baac"}
Oct 30 21:10:45.274524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:45.274116    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" event={"ID":"cae247f2-d22b-4593-9935-071384bb7be9","Type":"ContainerStarted","Data":"3b5445c55551e1210c7a2446aaf80bc023b741744cf64cd8ec4806757f6e5efb"}
Oct 30 21:10:45.297483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:45.297432    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podStartSLOduration=2.297417993 podStartE2EDuration="2.297417993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:45.295816106 +0000 UTC m=+553.645543501" watchObservedRunningTime="2026-10-30 21:10:45.297417993 +0000 UTC m=+553.647145389"
Oct 30 21:10:46.101447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:46.101412    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:46.102974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:46.102951    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 21:10:46.103090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:46.102999    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 21:10:47.101416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:47.101367    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 21:10:47.101841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:47.101446    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 21:10:48.101796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:48.101763    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 21:10:48.102208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:48.101817    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 21:10:49.102099 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:49.102059    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 21:10:49.102489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:49.102113    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 21:10:50.101535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:50.101496    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 21:10:50.101715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:50.101553    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 21:10:51.101819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:51.101778    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 21:10:51.102363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:51.101842    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 21:10:52.916498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:52.916462    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:52.917039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:52.916889    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:52.917946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:52.917924    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48"
Oct 30 21:10:55.013641 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:10:55.013601    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:10:58.793409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.793379    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"]
Oct 30 21:10:58.797497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.797469    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.811643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.811616    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"]
Oct 30 21:10:58.917364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.917310    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7911991f-b0ac-4181-ac75-1b25bacca07f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.917364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.917370    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.917645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.917495    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.917645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.917567    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.917645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.917601    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.917805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.917730    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thgpt\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-kube-api-access-thgpt\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.917866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.917840    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.917921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.917870    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:58.918028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:58.918004    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.018965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.018906    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.018965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.018966    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019014    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019055    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7911991f-b0ac-4181-ac75-1b25bacca07f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019175    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019224    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019272    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019302    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019338    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thgpt\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-kube-api-access-thgpt\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019464    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019652    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019758    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.019920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019898    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7911991f-b0ac-4181-ac75-1b25bacca07f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.020009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.019991    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.021415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.021391    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.021836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.021817    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.027493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.027468    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.027741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.027719    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thgpt\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-kube-api-access-thgpt\") pod \"maas-default-gateway-openshift-default-75bf867b6c-wlw8q\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.113595 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.113513    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:10:59.394029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.394006    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"]
Oct 30 21:10:59.396550 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:10:59.396524    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7911991f_b0ac_4181_ac75_1b25bacca07f.slice/crio-41285996360a7c51f3015e699d77a24eea39e2f550c2a0ccdb979b18f5631416 WatchSource:0}: Error finding container 41285996360a7c51f3015e699d77a24eea39e2f550c2a0ccdb979b18f5631416: Status 404 returned error can't find the container with id 41285996360a7c51f3015e699d77a24eea39e2f550c2a0ccdb979b18f5631416
Oct 30 21:10:59.399247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.399212    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:59.399346 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.399278    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:10:59.399346 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:10:59.399307    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:00.360716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:00.360653    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" event={"ID":"7911991f-b0ac-4181-ac75-1b25bacca07f","Type":"ContainerStarted","Data":"7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb"}
Oct 30 21:11:00.360716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:00.360721    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" event={"ID":"7911991f-b0ac-4181-ac75-1b25bacca07f","Type":"ContainerStarted","Data":"41285996360a7c51f3015e699d77a24eea39e2f550c2a0ccdb979b18f5631416"}
Oct 30 21:11:00.384117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:00.384069    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podStartSLOduration=2.384055509 podStartE2EDuration="2.384055509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:00.381867814 +0000 UTC m=+568.731595209" watchObservedRunningTime="2026-10-30 21:11:00.384055509 +0000 UTC m=+568.733782905"
Oct 30 21:11:01.122297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:01.122008    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:11:01.127699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:01.124410    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:11:01.127699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:01.124502    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 21:11:02.129711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:02.129338    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:11:02.129711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:02.129406    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 21:11:03.122697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:03.117977    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:11:03.122697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:03.118051    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 21:11:04.134694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:04.133118    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:11:04.134694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:04.133196    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 21:11:05.105205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.105176    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"]
Oct 30 21:11:05.110248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.110223    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.140114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.140078    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:11:05.142824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.142787    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 21:11:05.143354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.141485    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6eedd82b-openshift-default-dockercfg-df765\""
Oct 30 21:11:05.145942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.145914    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"]
Oct 30 21:11:05.191444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.191405    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-credential-socket\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.191627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.191546    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-socket\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.191627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.191587    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9516543-6c63-449d-8618-72df184b0a2f-istiod-ca-cert\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.191790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.191749    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-data\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.191855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.191797    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9516543-6c63-449d-8618-72df184b0a2f-istio-podinfo\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.191908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.191872    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-istio-token\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.191952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.191938    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w82f\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-kube-api-access-4w82f\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.191999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.191977    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-envoy\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.192131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.192107    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-certs\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293059    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-credential-socket\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293111    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-socket\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293139    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9516543-6c63-449d-8618-72df184b0a2f-istiod-ca-cert\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293184    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-data\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293213    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9516543-6c63-449d-8618-72df184b0a2f-istio-podinfo\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293332    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-istio-token\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293367    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w82f\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-kube-api-access-4w82f\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293393    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-envoy\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293438    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-certs\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293541    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-credential-socket\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293600    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-socket\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.293868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.293827    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-certs\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.294142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.294117    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-data\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.294396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.294376    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9516543-6c63-449d-8618-72df184b0a2f-istiod-ca-cert\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.295957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.295933    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-envoy\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.296260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.296242    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9516543-6c63-449d-8618-72df184b0a2f-istio-podinfo\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.304635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.304611    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-istio-token\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.305069 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.305050    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w82f\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-kube-api-access-4w82f\") pod \"e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") " pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.457661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.457621    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:05.512700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.512007    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"]
Oct 30 21:11:05.517148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.517131    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.532920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.532240    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"]
Oct 30 21:11:05.553486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.553296    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e239cab2-gw-openshift-default-dockercfg-7thg5\""
Oct 30 21:11:05.597385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597348    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-envoy\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.597534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597399    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-data\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.597534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597454    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-credential-socket\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.597534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597509    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-certs\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.597645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597541    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-podinfo\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.597645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597600    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-token\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.597645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597639    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-socket\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.597894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597780    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvjx\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-kube-api-access-rpvjx\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.597894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.597829    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789b6512-9b4b-4504-bd33-630e72d8ba8d-istiod-ca-cert\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.698433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698401    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-token\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.698433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698448    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-socket\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.698887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698778    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvjx\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-kube-api-access-rpvjx\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.698887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698816    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789b6512-9b4b-4504-bd33-630e72d8ba8d-istiod-ca-cert\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.698887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698864    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-envoy\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.698887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698881    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-data\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.699127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698916    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-credential-socket\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.699127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698920    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-socket\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.699127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698948    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-certs\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.699127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.698967    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-podinfo\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.699528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.699505    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-data\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.699889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.699863    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-certs\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.700001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.699982    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-credential-socket\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.700228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.700194    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789b6512-9b4b-4504-bd33-630e72d8ba8d-istiod-ca-cert\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.701316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.701294    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-envoy\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.702722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.702696    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-podinfo\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.706826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.706797    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvjx\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-kube-api-access-rpvjx\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.707230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.707206    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-token\") pod \"e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") " pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:05.752018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.751986    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"]
Oct 30 21:11:05.753388 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:05.753363    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9516543_6c63_449d_8618_72df184b0a2f.slice/crio-8f28de7aa10594fc1f6d9405928b5396eb4843d13dc8722686bbd1c91a67f7c7 WatchSource:0}: Error finding container 8f28de7aa10594fc1f6d9405928b5396eb4843d13dc8722686bbd1c91a67f7c7: Status 404 returned error can't find the container with id 8f28de7aa10594fc1f6d9405928b5396eb4843d13dc8722686bbd1c91a67f7c7
Oct 30 21:11:05.755944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.755895    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:05.756060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.755981    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:05.756060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.756034    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:05.851455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:05.851417    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:06.114493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.114399    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:11:06.114493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.114457    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 21:11:06.145314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.145285    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"]
Oct 30 21:11:06.146956 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:06.146929    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789b6512_9b4b_4504_bd33_630e72d8ba8d.slice/crio-a07e929d9bbbc6744c291a2d6f7b99408e90a12fe05accd086f39ea95fb246be WatchSource:0}: Error finding container a07e929d9bbbc6744c291a2d6f7b99408e90a12fe05accd086f39ea95fb246be: Status 404 returned error can't find the container with id a07e929d9bbbc6744c291a2d6f7b99408e90a12fe05accd086f39ea95fb246be
Oct 30 21:11:06.149312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.149278    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:06.149398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.149347    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:06.149398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.149377    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:06.397136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.397029    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" event={"ID":"d9516543-6c63-449d-8618-72df184b0a2f","Type":"ContainerStarted","Data":"ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58"}
Oct 30 21:11:06.397136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.397090    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" event={"ID":"d9516543-6c63-449d-8618-72df184b0a2f","Type":"ContainerStarted","Data":"8f28de7aa10594fc1f6d9405928b5396eb4843d13dc8722686bbd1c91a67f7c7"}
Oct 30 21:11:06.398801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.398774    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" event={"ID":"789b6512-9b4b-4504-bd33-630e72d8ba8d","Type":"ContainerStarted","Data":"48b36fe34c083d6843777295411380ff4035473224733b53af10b3cbe4690b44"}
Oct 30 21:11:06.398963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.398819    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" event={"ID":"789b6512-9b4b-4504-bd33-630e72d8ba8d","Type":"ContainerStarted","Data":"a07e929d9bbbc6744c291a2d6f7b99408e90a12fe05accd086f39ea95fb246be"}
Oct 30 21:11:06.428044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.427153    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" podStartSLOduration=1.427134266 podStartE2EDuration="1.427134266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:06.426266346 +0000 UTC m=+574.775993743" watchObservedRunningTime="2026-10-30 21:11:06.427134266 +0000 UTC m=+574.776861663"
Oct 30 21:11:06.458070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.458031    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:06.459167 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.459115    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podStartSLOduration=1.459099417 podStartE2EDuration="1.459099417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:06.457413193 +0000 UTC m=+574.807140589" watchObservedRunningTime="2026-10-30 21:11:06.459099417 +0000 UTC m=+574.808826814"
Oct 30 21:11:06.464243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.464220    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:06.852584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:06.852550    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:07.406770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:07.406736    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:07.407607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:07.407587    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:07.856934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:07.856855    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:07.913128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:07.913099    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:11:07.913541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:07.913520    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:11:07.914614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:07.914592    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:11:08.411749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:08.411717    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:08.412453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:08.412438    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:08.973752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:08.973719    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"]
Oct 30 21:11:08.977558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:08.977542    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:08.982891 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:08.982859    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e239cab2-dockercfg-rv2l2\""
Oct 30 21:11:08.983036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:08.982924    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e239cab2\""
Oct 30 21:11:08.987044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:08.987017    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"]
Oct 30 21:11:09.002651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.002624    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"]
Oct 30 21:11:09.006340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.006321    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:09.014123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.014100    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"]
Oct 30 21:11:09.135023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.134990    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgt4\" (UniqueName: \"kubernetes.io/projected/ec68ab05-108c-40d9-8323-bf953c3b7f1f-kube-api-access-fqgt4\") pod \"payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf\" (UID: \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:09.135228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.135065    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95ch\" (UniqueName: \"kubernetes.io/projected/c413f067-6fe2-4579-8ebe-9524c3f97d27-kube-api-access-h95ch\") pod \"payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf\" (UID: \"c413f067-6fe2-4579-8ebe-9524c3f97d27\") " pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:09.135228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.135207    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec68ab05-108c-40d9-8323-bf953c3b7f1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf\" (UID: \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:09.135336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.135265    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413f067-6fe2-4579-8ebe-9524c3f97d27-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf\" (UID: \"c413f067-6fe2-4579-8ebe-9524c3f97d27\") " pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:09.236736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.236598    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95ch\" (UniqueName: \"kubernetes.io/projected/c413f067-6fe2-4579-8ebe-9524c3f97d27-kube-api-access-h95ch\") pod \"payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf\" (UID: \"c413f067-6fe2-4579-8ebe-9524c3f97d27\") " pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:09.236736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.236713    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec68ab05-108c-40d9-8323-bf953c3b7f1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf\" (UID: \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:09.236970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.236759    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413f067-6fe2-4579-8ebe-9524c3f97d27-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf\" (UID: \"c413f067-6fe2-4579-8ebe-9524c3f97d27\") " pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:09.236970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.236863    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgt4\" (UniqueName: \"kubernetes.io/projected/ec68ab05-108c-40d9-8323-bf953c3b7f1f-kube-api-access-fqgt4\") pod \"payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf\" (UID: \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:09.237492 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.237468    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413f067-6fe2-4579-8ebe-9524c3f97d27-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf\" (UID: \"c413f067-6fe2-4579-8ebe-9524c3f97d27\") " pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:09.237580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.237499    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec68ab05-108c-40d9-8323-bf953c3b7f1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf\" (UID: \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:09.255248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.255202    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95ch\" (UniqueName: \"kubernetes.io/projected/c413f067-6fe2-4579-8ebe-9524c3f97d27-kube-api-access-h95ch\") pod \"payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf\" (UID: \"c413f067-6fe2-4579-8ebe-9524c3f97d27\") " pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:09.255402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.255316    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgt4\" (UniqueName: \"kubernetes.io/projected/ec68ab05-108c-40d9-8323-bf953c3b7f1f-kube-api-access-fqgt4\") pod \"payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf\" (UID: \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:09.260640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.260612    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"]
Oct 30 21:11:09.265031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.265009    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.277308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.277268    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"]
Oct 30 21:11:09.278429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.278408    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rx79k\""
Oct 30 21:11:09.278548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.278500    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e239cab2-serving-cert\""
Oct 30 21:11:09.278644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.278549    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:11:09.278978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.278960    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:11:09.279077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.279053    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:11:09.296182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.296153    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:09.323189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.323163    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:09.343715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.337921    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-metrics-certs\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.343715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.337994    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9kg\" (UniqueName: \"kubernetes.io/projected/23585a12-f5b6-4256-ac3a-53ca30b70e5f-kube-api-access-lg9kg\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.343715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.338026    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23585a12-f5b6-4256-ac3a-53ca30b70e5f-openshift-service-ca\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.343715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.338075    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-maas-api-tls\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.439716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.439236    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-metrics-certs\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.439716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.439292    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9kg\" (UniqueName: \"kubernetes.io/projected/23585a12-f5b6-4256-ac3a-53ca30b70e5f-kube-api-access-lg9kg\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.439716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.439311    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23585a12-f5b6-4256-ac3a-53ca30b70e5f-openshift-service-ca\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.439716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.439345    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-maas-api-tls\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.440264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.440246    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23585a12-f5b6-4256-ac3a-53ca30b70e5f-openshift-service-ca\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.442355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.442336    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-maas-api-tls\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.442446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.442378    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-metrics-certs\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.450874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.450847    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9kg\" (UniqueName: \"kubernetes.io/projected/23585a12-f5b6-4256-ac3a-53ca30b70e5f-kube-api-access-lg9kg\") pod \"maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.585685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.585593    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:09.591550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.591470    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"]
Oct 30 21:11:09.649033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.649007    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"]
Oct 30 21:11:09.669108 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:09.669077    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc413f067_6fe2_4579_8ebe_9524c3f97d27.slice/crio-edc28ceadec54217b94c5da5f0b4ec8efcec28f2d9b20286cdb4652abf95a672 WatchSource:0}: Error finding container edc28ceadec54217b94c5da5f0b4ec8efcec28f2d9b20286cdb4652abf95a672: Status 404 returned error can't find the container with id edc28ceadec54217b94c5da5f0b4ec8efcec28f2d9b20286cdb4652abf95a672
Oct 30 21:11:09.890097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:09.890073    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"]
Oct 30 21:11:09.891256 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:09.891228    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23585a12_f5b6_4256_ac3a_53ca30b70e5f.slice/crio-eb1aa5a2f3e47e514384dd40082bc787f8caf8ac51ec3d2a6bfb8a34f8334b6b WatchSource:0}: Error finding container eb1aa5a2f3e47e514384dd40082bc787f8caf8ac51ec3d2a6bfb8a34f8334b6b: Status 404 returned error can't find the container with id eb1aa5a2f3e47e514384dd40082bc787f8caf8ac51ec3d2a6bfb8a34f8334b6b
Oct 30 21:11:10.432623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.432586    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk" event={"ID":"23585a12-f5b6-4256-ac3a-53ca30b70e5f","Type":"ContainerStarted","Data":"eb1aa5a2f3e47e514384dd40082bc787f8caf8ac51ec3d2a6bfb8a34f8334b6b"}
Oct 30 21:11:10.434419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.434394    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf" event={"ID":"c413f067-6fe2-4579-8ebe-9524c3f97d27","Type":"ContainerStarted","Data":"6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511"}
Oct 30 21:11:10.434569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.434437    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf" event={"ID":"c413f067-6fe2-4579-8ebe-9524c3f97d27","Type":"ContainerStarted","Data":"edc28ceadec54217b94c5da5f0b4ec8efcec28f2d9b20286cdb4652abf95a672"}
Oct 30 21:11:10.434569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.434491    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:10.436139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.436113    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf" event={"ID":"ec68ab05-108c-40d9-8323-bf953c3b7f1f","Type":"ContainerStarted","Data":"abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825"}
Oct 30 21:11:10.436263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.436155    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf" event={"ID":"ec68ab05-108c-40d9-8323-bf953c3b7f1f","Type":"ContainerStarted","Data":"b6b9d2605c294dd19a62a8c70c16a0847f67032dfc8176388c14b0749d0e9309"}
Oct 30 21:11:10.436330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.436315    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:10.459465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.459414    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf" podStartSLOduration=2.4594 podStartE2EDuration="2.4594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:10.457429066 +0000 UTC m=+578.807156461" watchObservedRunningTime="2026-10-30 21:11:10.4594 +0000 UTC m=+578.809127388"
Oct 30 21:11:10.478594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.478539    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf" podStartSLOduration=2.478521241 podStartE2EDuration="2.478521241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:10.476726054 +0000 UTC m=+578.826453447" watchObservedRunningTime="2026-10-30 21:11:10.478521241 +0000 UTC m=+578.828248638"
Oct 30 21:11:10.539995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.539972    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"]
Oct 30 21:11:10.543618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.543602    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:10.556707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.556644    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6eedd82b\""
Oct 30 21:11:10.556707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.556654    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6eedd82b-dockercfg-sps7p\""
Oct 30 21:11:10.559472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.559452    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"]
Oct 30 21:11:10.666545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.666267    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvqs\" (UniqueName: \"kubernetes.io/projected/1a69c553-0696-4fd3-aa8a-e3c722811e1f-kube-api-access-zlvqs\") pod \"payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9\" (UID: \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:10.666545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.666397    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69c553-0696-4fd3-aa8a-e3c722811e1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9\" (UID: \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:10.768002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.767892    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69c553-0696-4fd3-aa8a-e3c722811e1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9\" (UID: \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:10.768149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.768027    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvqs\" (UniqueName: \"kubernetes.io/projected/1a69c553-0696-4fd3-aa8a-e3c722811e1f-kube-api-access-zlvqs\") pod \"payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9\" (UID: \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:10.768750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.768729    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69c553-0696-4fd3-aa8a-e3c722811e1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9\" (UID: \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:10.776510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.776484    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvqs\" (UniqueName: \"kubernetes.io/projected/1a69c553-0696-4fd3-aa8a-e3c722811e1f-kube-api-access-zlvqs\") pod \"payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9\" (UID: \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:10.857851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:10.857821    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:11.156730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.156705    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"]
Oct 30 21:11:11.157715 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:11.157684    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a69c553_0696_4fd3_aa8a_e3c722811e1f.slice/crio-751e8b13d5de5a056f24d4a5e5674e7393dacce73c3642e9d59fc9f9e33b36c2 WatchSource:0}: Error finding container 751e8b13d5de5a056f24d4a5e5674e7393dacce73c3642e9d59fc9f9e33b36c2: Status 404 returned error can't find the container with id 751e8b13d5de5a056f24d4a5e5674e7393dacce73c3642e9d59fc9f9e33b36c2
Oct 30 21:11:11.445083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.444987    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9" event={"ID":"1a69c553-0696-4fd3-aa8a-e3c722811e1f","Type":"ContainerStarted","Data":"1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a"}
Oct 30 21:11:11.445083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.445053    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"]
Oct 30 21:11:11.449052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.449028    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9" event={"ID":"1a69c553-0696-4fd3-aa8a-e3c722811e1f","Type":"ContainerStarted","Data":"751e8b13d5de5a056f24d4a5e5674e7393dacce73c3642e9d59fc9f9e33b36c2"}
Oct 30 21:11:11.449155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.449066    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:11.449155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.449091    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:11.449155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.449106    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk" event={"ID":"23585a12-f5b6-4256-ac3a-53ca30b70e5f","Type":"ContainerStarted","Data":"ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693"}
Oct 30 21:11:11.449254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.449165    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:11.463054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.463031    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"]
Oct 30 21:11:11.468750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.468726    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-111fd691\""
Oct 30 21:11:11.469510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.469457    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9" podStartSLOduration=1.4694419060000001 podStartE2EDuration="1.469441906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:11.467607654 +0000 UTC m=+579.817335061" watchObservedRunningTime="2026-10-30 21:11:11.469441906 +0000 UTC m=+579.819169302"
Oct 30 21:11:11.490054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.490003    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk" podStartSLOduration=1.9196037860000001 podStartE2EDuration="2.489990134s" totalImagesPullingTime="570.386348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:11.488870927 +0000 UTC m=+579.838598325" watchObservedRunningTime="2026-10-30 21:11:11.489990134 +0000 UTC m=+579.839717530"
Oct 30 21:11:11.519937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.519907    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"]
Oct 30 21:11:11.523648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.523631    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:11.530078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.530048    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-111fd691-dockercfg-gbbsv\""
Oct 30 21:11:11.534063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.533757    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"]
Oct 30 21:11:11.576235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.576193    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a9d602-2259-47e3-bf24-de836933255a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp\" (UID: \"f2a9d602-2259-47e3-bf24-de836933255a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:11.576435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.576399    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwxjf\" (UniqueName: \"kubernetes.io/projected/f2a9d602-2259-47e3-bf24-de836933255a-kube-api-access-qwxjf\") pod \"payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp\" (UID: \"f2a9d602-2259-47e3-bf24-de836933255a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:11.677398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.677358    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fff94e-995e-4eee-bc7a-67d66ed68b50-plugins-config-volume\") pod \"payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq\" (UID: \"17fff94e-995e-4eee-bc7a-67d66ed68b50\") " pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:11.677612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.677409    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwxjf\" (UniqueName: \"kubernetes.io/projected/f2a9d602-2259-47e3-bf24-de836933255a-kube-api-access-qwxjf\") pod \"payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp\" (UID: \"f2a9d602-2259-47e3-bf24-de836933255a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:11.677612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.677596    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt4vq\" (UniqueName: \"kubernetes.io/projected/17fff94e-995e-4eee-bc7a-67d66ed68b50-kube-api-access-jt4vq\") pod \"payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq\" (UID: \"17fff94e-995e-4eee-bc7a-67d66ed68b50\") " pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:11.677736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.677652    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a9d602-2259-47e3-bf24-de836933255a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp\" (UID: \"f2a9d602-2259-47e3-bf24-de836933255a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:11.678349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.678331    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a9d602-2259-47e3-bf24-de836933255a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp\" (UID: \"f2a9d602-2259-47e3-bf24-de836933255a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:11.689047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.689021    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwxjf\" (UniqueName: \"kubernetes.io/projected/f2a9d602-2259-47e3-bf24-de836933255a-kube-api-access-qwxjf\") pod \"payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp\" (UID: \"f2a9d602-2259-47e3-bf24-de836933255a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:11.764777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.764691    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:11.779306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.779276    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fff94e-995e-4eee-bc7a-67d66ed68b50-plugins-config-volume\") pod \"payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq\" (UID: \"17fff94e-995e-4eee-bc7a-67d66ed68b50\") " pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:11.779465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.779447    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt4vq\" (UniqueName: \"kubernetes.io/projected/17fff94e-995e-4eee-bc7a-67d66ed68b50-kube-api-access-jt4vq\") pod \"payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq\" (UID: \"17fff94e-995e-4eee-bc7a-67d66ed68b50\") " pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:11.780027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.780007    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fff94e-995e-4eee-bc7a-67d66ed68b50-plugins-config-volume\") pod \"payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq\" (UID: \"17fff94e-995e-4eee-bc7a-67d66ed68b50\") " pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:11.791324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.791297    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt4vq\" (UniqueName: \"kubernetes.io/projected/17fff94e-995e-4eee-bc7a-67d66ed68b50-kube-api-access-jt4vq\") pod \"payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq\" (UID: \"17fff94e-995e-4eee-bc7a-67d66ed68b50\") " pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:11.840140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:11.840099    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:12.062881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.062846    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"]
Oct 30 21:11:12.064229 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:12.064199    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a9d602_2259_47e3_bf24_de836933255a.slice/crio-1dd383544abc82c356295e627088253ebe9ff29dba6545e2dbc3e2999f744832 WatchSource:0}: Error finding container 1dd383544abc82c356295e627088253ebe9ff29dba6545e2dbc3e2999f744832: Status 404 returned error can't find the container with id 1dd383544abc82c356295e627088253ebe9ff29dba6545e2dbc3e2999f744832
Oct 30 21:11:12.128739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.128580    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"]
Oct 30 21:11:12.129208 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:12.129175    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fff94e_995e_4eee_bc7a_67d66ed68b50.slice/crio-2497f139571a354a04ad0e1bfe040837ab4aa203cdf17e5c74c08652291a7485 WatchSource:0}: Error finding container 2497f139571a354a04ad0e1bfe040837ab4aa203cdf17e5c74c08652291a7485: Status 404 returned error can't find the container with id 2497f139571a354a04ad0e1bfe040837ab4aa203cdf17e5c74c08652291a7485
Oct 30 21:11:12.307589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.307548    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:11:12.454744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.454704    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq" event={"ID":"17fff94e-995e-4eee-bc7a-67d66ed68b50","Type":"ContainerStarted","Data":"88bab4e7bc3cc4e0ffdcacf883f6d28546971ab2db5cb67b0a6df3e1bc30a423"}
Oct 30 21:11:12.454942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.454767    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq" event={"ID":"17fff94e-995e-4eee-bc7a-67d66ed68b50","Type":"ContainerStarted","Data":"2497f139571a354a04ad0e1bfe040837ab4aa203cdf17e5c74c08652291a7485"}
Oct 30 21:11:12.454942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.454916    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:12.456560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.456522    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp" event={"ID":"f2a9d602-2259-47e3-bf24-de836933255a","Type":"ContainerStarted","Data":"b8c027fa7278aa260f91e8147b92b147f7b83314d403e7557a936ddf9a828fa2"}
Oct 30 21:11:12.456719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.456572    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp" event={"ID":"f2a9d602-2259-47e3-bf24-de836933255a","Type":"ContainerStarted","Data":"1dd383544abc82c356295e627088253ebe9ff29dba6545e2dbc3e2999f744832"}
Oct 30 21:11:12.456719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.456621    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:12.476311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.476259    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq" podStartSLOduration=1.476243389 podStartE2EDuration="1.476243389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:12.47458039 +0000 UTC m=+580.824307785" watchObservedRunningTime="2026-10-30 21:11:12.476243389 +0000 UTC m=+580.825970786"
Oct 30 21:11:12.519775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.519696    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp" podStartSLOduration=1.51965468 podStartE2EDuration="1.51965468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:12.493812055 +0000 UTC m=+580.843539454" watchObservedRunningTime="2026-10-30 21:11:12.51965468 +0000 UTC m=+580.869382076"
Oct 30 21:11:12.965772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.965742    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"]
Oct 30 21:11:12.970662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.970637    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:12.981094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.981068    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"]
Oct 30 21:11:12.987118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:12.987095    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6eedd82b-serving-cert\""
Oct 30 21:11:13.094857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.094816    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnqr\" (UniqueName: \"kubernetes.io/projected/0d18b6f4-20eb-4162-8202-063ba87f5a43-kube-api-access-4cnqr\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.095218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.094874    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-metrics-certs\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.095218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.094946    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-maas-api-tls\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.095218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.094998    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d18b6f4-20eb-4162-8202-063ba87f5a43-openshift-service-ca\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.196616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.196523    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-maas-api-tls\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.196825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.196644    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d18b6f4-20eb-4162-8202-063ba87f5a43-openshift-service-ca\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.196980 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.196895    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnqr\" (UniqueName: \"kubernetes.io/projected/0d18b6f4-20eb-4162-8202-063ba87f5a43-kube-api-access-4cnqr\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.197053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.197008    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-metrics-certs\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.197444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.197419    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d18b6f4-20eb-4162-8202-063ba87f5a43-openshift-service-ca\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.199881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.199859    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-maas-api-tls\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.200161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.200136    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-metrics-certs\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.206749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.206726    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnqr\" (UniqueName: \"kubernetes.io/projected/0d18b6f4-20eb-4162-8202-063ba87f5a43-kube-api-access-4cnqr\") pod \"maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.295154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.295076    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:13.611020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:13.610992    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"]
Oct 30 21:11:13.612338 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:13.612312    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d18b6f4_20eb_4162_8202_063ba87f5a43.slice/crio-56047e34231c94064aa459701ff9099a871115800600796e9fbb6a86beb0273f WatchSource:0}: Error finding container 56047e34231c94064aa459701ff9099a871115800600796e9fbb6a86beb0273f: Status 404 returned error can't find the container with id 56047e34231c94064aa459701ff9099a871115800600796e9fbb6a86beb0273f
Oct 30 21:11:14.478939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:14.478900    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8" event={"ID":"0d18b6f4-20eb-4162-8202-063ba87f5a43","Type":"ContainerStarted","Data":"d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283"}
Oct 30 21:11:14.479113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:14.478983    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8" event={"ID":"0d18b6f4-20eb-4162-8202-063ba87f5a43","Type":"ContainerStarted","Data":"56047e34231c94064aa459701ff9099a871115800600796e9fbb6a86beb0273f"}
Oct 30 21:11:14.479113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:14.479011    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:14.502185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:14.502130    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8" podStartSLOduration=2.013004432 podStartE2EDuration="2.502114338s" totalImagesPullingTime="489.109906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:14.499230878 +0000 UTC m=+582.848958275" watchObservedRunningTime="2026-10-30 21:11:14.502114338 +0000 UTC m=+582.851841735"
Oct 30 21:11:17.469879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:17.469830    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:19.095364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:19.095327    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:11:20.471027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.470985    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"]
Oct 30 21:11:20.477043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.477004    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.495087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.492264    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5ee9aa6f-serving-cert\""
Oct 30 21:11:20.495835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.495777    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"]
Oct 30 21:11:20.544296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.544266    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:20.548032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.547998    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a20f96-68c1-440d-ae36-653d4afa8684-openshift-service-ca\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.548310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.548285    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-metrics-certs\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.548509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.548467    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jlm2\" (UniqueName: \"kubernetes.io/projected/12a20f96-68c1-440d-ae36-653d4afa8684-kube-api-access-9jlm2\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.548736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.548716    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.650388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.650346    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.650575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.650476    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a20f96-68c1-440d-ae36-653d4afa8684-openshift-service-ca\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.650652 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:20.650570    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-serving-cert: secret "maas-api-e2e-derive-5ee9aa6f-serving-cert" not found
Oct 30 21:11:20.650652 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:20.650637    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls podName:12a20f96-68c1-440d-ae36-653d4afa8684 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:21.150616069 +0000 UTC m=+589.500343448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls") pod "maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz" (UID: "12a20f96-68c1-440d-ae36-653d4afa8684") : secret "maas-api-e2e-derive-5ee9aa6f-serving-cert" not found
Oct 30 21:11:20.650825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.650706    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-metrics-certs\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.650825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.650789    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jlm2\" (UniqueName: \"kubernetes.io/projected/12a20f96-68c1-440d-ae36-653d4afa8684-kube-api-access-9jlm2\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.651706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.651625    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a20f96-68c1-440d-ae36-653d4afa8684-openshift-service-ca\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.653440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.653415    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-metrics-certs\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:20.659450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:20.659403    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jlm2\" (UniqueName: \"kubernetes.io/projected/12a20f96-68c1-440d-ae36-653d4afa8684-kube-api-access-9jlm2\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:21.156505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:21.156432    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:21.159128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:21.159096    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls\") pod \"maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:21.408586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:21.408497    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:21.450657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:21.450556    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:21.451060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:21.451039    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:21.741895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:21.741859    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"]
Oct 30 21:11:21.743280 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:21.743245    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a20f96_68c1_440d_ae36_653d4afa8684.slice/crio-a68540afd1e89948f56474957369f606e332d6ce91056f1a4f1e8640dd3b5a31 WatchSource:0}: Error finding container a68540afd1e89948f56474957369f606e332d6ce91056f1a4f1e8640dd3b5a31: Status 404 returned error can't find the container with id a68540afd1e89948f56474957369f606e332d6ce91056f1a4f1e8640dd3b5a31
Oct 30 21:11:22.585117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:22.585034    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz" event={"ID":"12a20f96-68c1-440d-ae36-653d4afa8684","Type":"ContainerStarted","Data":"bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad"}
Oct 30 21:11:22.585289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:22.585131    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz" event={"ID":"12a20f96-68c1-440d-ae36-653d4afa8684","Type":"ContainerStarted","Data":"a68540afd1e89948f56474957369f606e332d6ce91056f1a4f1e8640dd3b5a31"}
Oct 30 21:11:22.585560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:22.585537    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:22.605034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:22.604981    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz" podStartSLOduration=2.049146499 podStartE2EDuration="2.604966582s" totalImagesPullingTime="555.820083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:22.603216664 +0000 UTC m=+590.952944087" watchObservedRunningTime="2026-10-30 21:11:22.604966582 +0000 UTC m=+590.954693978"
Oct 30 21:11:23.466650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:23.466557    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:23.467143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:23.466730    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:23.467143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:23.466794    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:23.695553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:23.694932    4633 patch_prober.go:29] interesting pod/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:53718->10.133.0.63:15021: read: connection reset by peer" start-of-body=
Oct 30 21:11:23.695553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:23.694986    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:53718->10.133.0.63:15021: read: connection reset by peer"
Oct 30 21:11:24.604984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:24.604950    4633 generic.go:356] "Generic (PLEG): container finished" podID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerID="48b36fe34c083d6843777295411380ff4035473224733b53af10b3cbe4690b44" exitCode=0
Oct 30 21:11:24.604984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:24.605000    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" event={"ID":"789b6512-9b4b-4504-bd33-630e72d8ba8d","Type":"ContainerDied","Data":"48b36fe34c083d6843777295411380ff4035473224733b53af10b3cbe4690b44"}
Oct 30 21:11:24.605544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:24.605344    4633 scope.go:122] "RemoveContainer" containerID="48b36fe34c083d6843777295411380ff4035473224733b53af10b3cbe4690b44"
Oct 30 21:11:24.606593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:24.606557    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:24.606709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:24.606631    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:24.606709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:24.606692    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:25.020933 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:25.020887    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:11:25.649924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:25.649882    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" event={"ID":"789b6512-9b4b-4504-bd33-630e72d8ba8d","Type":"ContainerStarted","Data":"5e22fd1f5d8fa96b0046dd86abfbda9abc2a9bb12cdc56b6d18c3abe737a4626"}
Oct 30 21:11:25.852223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:25.852185    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:25.854090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:25.852499    4633 patch_prober.go:29] interesting pod/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:11:25.854090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:25.852542    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:11:25.908308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:25.908275    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:11:26.658166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:26.658130    4633 generic.go:356] "Generic (PLEG): container finished" podID="d9516543-6c63-449d-8618-72df184b0a2f" containerID="ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58" exitCode=0
Oct 30 21:11:26.658585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:26.658201    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" event={"ID":"d9516543-6c63-449d-8618-72df184b0a2f","Type":"ContainerDied","Data":"ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58"}
Oct 30 21:11:26.658692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:26.658659    4633 scope.go:122] "RemoveContainer" containerID="ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58"
Oct 30 21:11:26.659799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:26.659774    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:26.659857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:26.659836    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:26.659892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:26.659867    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:26.852117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:26.852081    4633 patch_prober.go:29] interesting pod/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:11:26.852284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:26.852127    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:11:27.670838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:27.670798    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" event={"ID":"d9516543-6c63-449d-8618-72df184b0a2f","Type":"ContainerStarted","Data":"002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f"}
Oct 30 21:11:27.860593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:27.860554    4633 patch_prober.go:29] interesting pod/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:11:27.860883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:27.860849    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:11:28.458165 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:28.458127    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:28.458517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:28.458495    4633 patch_prober.go:29] interesting pod/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:11:28.458718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:28.458658    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:11:28.601094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:28.601064    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:28.860213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:28.860125    4633 patch_prober.go:29] interesting pod/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:11:28.860841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:28.860812    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:11:29.458695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:29.458108    4633 patch_prober.go:29] interesting pod/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:11:29.458695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:29.458171    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:11:29.852191 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:29.852096    4633 patch_prober.go:29] interesting pod/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:11:29.859694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:29.852548    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:11:30.460697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:30.459850    4633 patch_prober.go:29] interesting pod/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:11:30.460697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:30.459921    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:11:30.852479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:30.852381    4633 patch_prober.go:29] interesting pod/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:11:30.852479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:30.852443    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:11:31.458201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.458170    4633 patch_prober.go:29] interesting pod/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:11:31.458391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.458227    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:11:31.858355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.857427    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:31.858355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.858166    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:31.859035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.859016    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:31.874234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.874213    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:11:31.876620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.876600    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:11:31.894191 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.894165    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:11:31.896465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:31.896433    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:11:32.458104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:32.458070    4633 patch_prober.go:29] interesting pod/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:11:32.458270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:32.458121    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:11:33.352559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:33.352521    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"]
Oct 30 21:11:33.367123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:33.367094    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"]
Oct 30 21:11:33.462437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:33.462409    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:33.462722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:33.462704    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:33.463408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:33.463389    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:36.017006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.016972    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"]
Oct 30 21:11:36.017454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.017179    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk" podUID="23585a12-f5b6-4256-ac3a-53ca30b70e5f" containerName="maas-api" containerID="cri-o://ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693" gracePeriod=30
Oct 30 21:11:36.339169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.339147    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:36.395063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.395034    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23585a12-f5b6-4256-ac3a-53ca30b70e5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23585a12-f5b6-4256-ac3a-53ca30b70e5f-openshift-service-ca\") pod \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") "
Oct 30 21:11:36.395249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.395133    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-metrics-certs\") pod \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") "
Oct 30 21:11:36.395249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.395159    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23585a12-f5b6-4256-ac3a-53ca30b70e5f-kube-api-access-lg9kg\" (UniqueName: \"kubernetes.io/projected/23585a12-f5b6-4256-ac3a-53ca30b70e5f-kube-api-access-lg9kg\") pod \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") "
Oct 30 21:11:36.395249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.395212    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-maas-api-tls\") pod \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\" (UID: \"23585a12-f5b6-4256-ac3a-53ca30b70e5f\") "
Oct 30 21:11:36.395480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.395458    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23585a12-f5b6-4256-ac3a-53ca30b70e5f-openshift-service-ca" pod "23585a12-f5b6-4256-ac3a-53ca30b70e5f" (UID: "23585a12-f5b6-4256-ac3a-53ca30b70e5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:36.395586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.395571    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23585a12-f5b6-4256-ac3a-53ca30b70e5f-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:36.397361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.397337    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-metrics-certs" pod "23585a12-f5b6-4256-ac3a-53ca30b70e5f" (UID: "23585a12-f5b6-4256-ac3a-53ca30b70e5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:36.397428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.397374    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-maas-api-tls" pod "23585a12-f5b6-4256-ac3a-53ca30b70e5f" (UID: "23585a12-f5b6-4256-ac3a-53ca30b70e5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:36.397467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.397430    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23585a12-f5b6-4256-ac3a-53ca30b70e5f-kube-api-access-lg9kg" pod "23585a12-f5b6-4256-ac3a-53ca30b70e5f" (UID: "23585a12-f5b6-4256-ac3a-53ca30b70e5f"). InnerVolumeSpecName "kube-api-access-lg9kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:36.497170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.497139    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:36.497170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.497167    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg9kg\" (UniqueName: \"kubernetes.io/projected/23585a12-f5b6-4256-ac3a-53ca30b70e5f-kube-api-access-lg9kg\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:36.497170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.497177    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23585a12-f5b6-4256-ac3a-53ca30b70e5f-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:36.675063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.675032    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"]
Oct 30 21:11:36.675264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.675231    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy" containerID="cri-o://5e22fd1f5d8fa96b0046dd86abfbda9abc2a9bb12cdc56b6d18c3abe737a4626" gracePeriod=30
Oct 30 21:11:36.736419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.736380    4633 generic.go:356] "Generic (PLEG): container finished" podID="23585a12-f5b6-4256-ac3a-53ca30b70e5f" containerID="ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693" exitCode=0
Oct 30 21:11:36.736609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.736425    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"
Oct 30 21:11:36.736609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.736492    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk" event={"ID":"23585a12-f5b6-4256-ac3a-53ca30b70e5f","Type":"ContainerDied","Data":"ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693"}
Oct 30 21:11:36.736609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.736539    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk" event={"ID":"23585a12-f5b6-4256-ac3a-53ca30b70e5f","Type":"ContainerDied","Data":"eb1aa5a2f3e47e514384dd40082bc787f8caf8ac51ec3d2a6bfb8a34f8334b6b"}
Oct 30 21:11:36.736609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.736568    4633 scope.go:122] "RemoveContainer" containerID="ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693"
Oct 30 21:11:36.753539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.753516    4633 scope.go:122] "RemoveContainer" containerID="ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693"
Oct 30 21:11:36.753898 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:36.753877    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693\": container with ID starting with ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693 not found: ID does not exist" containerID="ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693"
Oct 30 21:11:36.754018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.753905    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693"} err="failed to get container status \"ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693\": rpc error: code = NotFound desc = could not find container \"ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693\": container with ID starting with ca1c42d383ac49b958eac83c2d82b9f1e2b170dfba7057898612ea9811d6d693 not found: ID does not exist"
Oct 30 21:11:36.765025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.765000    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"]
Oct 30 21:11:36.770530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:36.770501    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e239cab2-596f98c879-k7dqk"]
Oct 30 21:11:37.790424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:37.790391    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23585a12-f5b6-4256-ac3a-53ca30b70e5f" path="/var/lib/kubelet/pods/23585a12-f5b6-4256-ac3a-53ca30b70e5f/volumes"
Oct 30 21:11:37.992227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:37.992195    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"]
Oct 30 21:11:37.992717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:37.992703    4633 cpu_manager.go:414] "containerMap: removing container" podUID="23585a12-f5b6-4256-ac3a-53ca30b70e5f" containerName="maas-api"
Oct 30 21:11:37.992783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:37.992719    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23585a12-f5b6-4256-ac3a-53ca30b70e5f" containerName="maas-api"
Oct 30 21:11:37.992825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:37.992820    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="23585a12-f5b6-4256-ac3a-53ca30b70e5f" containerName="maas-api"
Oct 30 21:11:37.997680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:37.997648    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.006155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.006134    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-05a342-openshift-default-dockercfg-vxlf4\""
Oct 30 21:11:38.009906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.009884    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"]
Oct 30 21:11:38.113952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.113859    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.113952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.113929    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778ea70-23d5-467d-bd3d-e034ef9327dc-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.114160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.113995    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.114160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.114022    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.114160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.114084    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g82t\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-kube-api-access-5g82t\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.114160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.114135    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.114297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.114175    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.114297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.114254    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.114363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.114305    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.214174    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.215899    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778ea70-23d5-467d-bd3d-e034ef9327dc-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.216656    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778ea70-23d5-467d-bd3d-e034ef9327dc-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.216948    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.216985    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.217013    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g82t\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-kube-api-access-5g82t\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.217061    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.217107    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.217166    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.217240    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.217276    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.217697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.217575    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.219751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.219428    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.219751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.219698    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.227729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.223872    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.228261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.228238    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.231237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.231209    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.239953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.239910    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g82t\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-kube-api-access-5g82t\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.240095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.240006    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.313788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.313751    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:38.353786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.353735    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf" podUID="c413f067-6fe2-4579-8ebe-9524c3f97d27" containerName="payload-processing" containerID="cri-o://6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511" gracePeriod=25
Oct 30 21:11:38.368400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.368357    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf" podUID="ec68ab05-108c-40d9-8323-bf953c3b7f1f" containerName="payload-pre-processing" containerID="cri-o://abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825" gracePeriod=25
Oct 30 21:11:38.601655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.601615    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"]
Oct 30 21:11:38.607349 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:38.607319    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5778ea70_23d5_467d_bd3d_e034ef9327dc.slice/crio-a1339a357d2a875d745e43d65532e3baa356e161b8357c592b2d9eac8276f40d WatchSource:0}: Error finding container a1339a357d2a875d745e43d65532e3baa356e161b8357c592b2d9eac8276f40d: Status 404 returned error can't find the container with id a1339a357d2a875d745e43d65532e3baa356e161b8357c592b2d9eac8276f40d
Oct 30 21:11:38.610581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.610538    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:38.610708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.610631    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:38.610770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.610701    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:38.642341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.642315    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:38.658008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.657979    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:38.722180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.722148    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec68ab05-108c-40d9-8323-bf953c3b7f1f-kube-api-access-fqgt4\" (UniqueName: \"kubernetes.io/projected/ec68ab05-108c-40d9-8323-bf953c3b7f1f-kube-api-access-fqgt4\") pod \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\" (UID: \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\") "
Oct 30 21:11:38.722366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.722291    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec68ab05-108c-40d9-8323-bf953c3b7f1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec68ab05-108c-40d9-8323-bf953c3b7f1f-plugins-config-volume\") pod \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\" (UID: \"ec68ab05-108c-40d9-8323-bf953c3b7f1f\") "
Oct 30 21:11:38.722446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.722401    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c413f067-6fe2-4579-8ebe-9524c3f97d27-kube-api-access-h95ch\" (UniqueName: \"kubernetes.io/projected/c413f067-6fe2-4579-8ebe-9524c3f97d27-kube-api-access-h95ch\") pod \"c413f067-6fe2-4579-8ebe-9524c3f97d27\" (UID: \"c413f067-6fe2-4579-8ebe-9524c3f97d27\") "
Oct 30 21:11:38.722790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.722755    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec68ab05-108c-40d9-8323-bf953c3b7f1f-plugins-config-volume" pod "ec68ab05-108c-40d9-8323-bf953c3b7f1f" (UID: "ec68ab05-108c-40d9-8323-bf953c3b7f1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:38.722927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.722825    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c413f067-6fe2-4579-8ebe-9524c3f97d27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413f067-6fe2-4579-8ebe-9524c3f97d27-plugins-config-volume\") pod \"c413f067-6fe2-4579-8ebe-9524c3f97d27\" (UID: \"c413f067-6fe2-4579-8ebe-9524c3f97d27\") "
Oct 30 21:11:38.723283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.723258    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c413f067-6fe2-4579-8ebe-9524c3f97d27-plugins-config-volume" pod "c413f067-6fe2-4579-8ebe-9524c3f97d27" (UID: "c413f067-6fe2-4579-8ebe-9524c3f97d27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:38.723528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.723508    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c413f067-6fe2-4579-8ebe-9524c3f97d27-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:38.723608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.723533    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec68ab05-108c-40d9-8323-bf953c3b7f1f-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:38.724755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.724725    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec68ab05-108c-40d9-8323-bf953c3b7f1f-kube-api-access-fqgt4" pod "ec68ab05-108c-40d9-8323-bf953c3b7f1f" (UID: "ec68ab05-108c-40d9-8323-bf953c3b7f1f"). InnerVolumeSpecName "kube-api-access-fqgt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:38.724854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.724777    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c413f067-6fe2-4579-8ebe-9524c3f97d27-kube-api-access-h95ch" pod "c413f067-6fe2-4579-8ebe-9524c3f97d27" (UID: "c413f067-6fe2-4579-8ebe-9524c3f97d27"). InnerVolumeSpecName "kube-api-access-h95ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:38.752456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.752364    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" event={"ID":"5778ea70-23d5-467d-bd3d-e034ef9327dc","Type":"ContainerStarted","Data":"b992455bdf9e7e4ab1d167e7d8d63c57cf084d05bbab877b475ead547ac2f1e3"}
Oct 30 21:11:38.752456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.752425    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" event={"ID":"5778ea70-23d5-467d-bd3d-e034ef9327dc","Type":"ContainerStarted","Data":"a1339a357d2a875d745e43d65532e3baa356e161b8357c592b2d9eac8276f40d"}
Oct 30 21:11:38.754213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.754176    4633 generic.go:356] "Generic (PLEG): container finished" podID="c413f067-6fe2-4579-8ebe-9524c3f97d27" containerID="6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511" exitCode=0
Oct 30 21:11:38.754213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.754210    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"
Oct 30 21:11:38.754402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.754248    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf" event={"ID":"c413f067-6fe2-4579-8ebe-9524c3f97d27","Type":"ContainerDied","Data":"6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511"}
Oct 30 21:11:38.754402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.754297    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf" event={"ID":"c413f067-6fe2-4579-8ebe-9524c3f97d27","Type":"ContainerDied","Data":"edc28ceadec54217b94c5da5f0b4ec8efcec28f2d9b20286cdb4652abf95a672"}
Oct 30 21:11:38.754402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.754325    4633 scope.go:122] "RemoveContainer" containerID="6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511"
Oct 30 21:11:38.755822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.755801    4633 generic.go:356] "Generic (PLEG): container finished" podID="ec68ab05-108c-40d9-8323-bf953c3b7f1f" containerID="abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825" exitCode=0
Oct 30 21:11:38.755945 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.755929    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf" event={"ID":"ec68ab05-108c-40d9-8323-bf953c3b7f1f","Type":"ContainerDied","Data":"abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825"}
Oct 30 21:11:38.756010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.755975    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf" event={"ID":"ec68ab05-108c-40d9-8323-bf953c3b7f1f","Type":"ContainerDied","Data":"b6b9d2605c294dd19a62a8c70c16a0847f67032dfc8176388c14b0749d0e9309"}
Oct 30 21:11:38.756048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.756040    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"
Oct 30 21:11:38.766776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.766757    4633 scope.go:122] "RemoveContainer" containerID="6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511"
Oct 30 21:11:38.767140 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:38.767111    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511\": container with ID starting with 6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511 not found: ID does not exist" containerID="6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511"
Oct 30 21:11:38.767248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.767150    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511"} err="failed to get container status \"6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511\": rpc error: code = NotFound desc = could not find container \"6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511\": container with ID starting with 6a65ed9330b75283df60355245ee87f44145bf74aaad99b495e71a10942c0511 not found: ID does not exist"
Oct 30 21:11:38.767248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.767172    4633 scope.go:122] "RemoveContainer" containerID="abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825"
Oct 30 21:11:38.781256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.781233    4633 scope.go:122] "RemoveContainer" containerID="abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825"
Oct 30 21:11:38.781641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.781592    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podStartSLOduration=1.781576624 podStartE2EDuration="1.781576624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:38.778444841 +0000 UTC m=+607.128172238" watchObservedRunningTime="2026-10-30 21:11:38.781576624 +0000 UTC m=+607.131304020"
Oct 30 21:11:38.781795 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:38.781654    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825\": container with ID starting with abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825 not found: ID does not exist" containerID="abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825"
Oct 30 21:11:38.781795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.781696    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825"} err="failed to get container status \"abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825\": rpc error: code = NotFound desc = could not find container \"abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825\": container with ID starting with abdfc6219c467e2121758f7102aadb0db3f20b59b91af538da821462a5867825 not found: ID does not exist"
Oct 30 21:11:38.795898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.795873    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"]
Oct 30 21:11:38.799730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.799704    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e239cab2-76579646d7-w5vbf"]
Oct 30 21:11:38.810140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.810119    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"]
Oct 30 21:11:38.817301 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.817277    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e239cab2-5f6d7898f9-2zqcf"]
Oct 30 21:11:38.825007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.824985    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqgt4\" (UniqueName: \"kubernetes.io/projected/ec68ab05-108c-40d9-8323-bf953c3b7f1f-kube-api-access-fqgt4\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:38.825103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:38.825090    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h95ch\" (UniqueName: \"kubernetes.io/projected/c413f067-6fe2-4579-8ebe-9524c3f97d27-kube-api-access-h95ch\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:39.314338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:39.314308    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:39.319432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:39.319410    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:39.754129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:39.754092    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"]
Oct 30 21:11:39.782477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:39.782448    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:39.797247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:39.797211    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c413f067-6fe2-4579-8ebe-9524c3f97d27" path="/var/lib/kubelet/pods/c413f067-6fe2-4579-8ebe-9524c3f97d27/volumes"
Oct 30 21:11:39.797796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:39.797728    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec68ab05-108c-40d9-8323-bf953c3b7f1f" path="/var/lib/kubelet/pods/ec68ab05-108c-40d9-8323-bf953c3b7f1f/volumes"
Oct 30 21:11:39.798087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:39.798072    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:11:40.909785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:40.909755    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"]
Oct 30 21:11:40.910168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:40.909942    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8" podUID="0d18b6f4-20eb-4162-8202-063ba87f5a43" containerName="maas-api" containerID="cri-o://d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283" gracePeriod=30
Oct 30 21:11:41.227421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.227391    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:41.255340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.255312    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d18b6f4-20eb-4162-8202-063ba87f5a43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d18b6f4-20eb-4162-8202-063ba87f5a43-openshift-service-ca\") pod \"0d18b6f4-20eb-4162-8202-063ba87f5a43\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") "
Oct 30 21:11:41.255806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.255755    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-metrics-certs\") pod \"0d18b6f4-20eb-4162-8202-063ba87f5a43\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") "
Oct 30 21:11:41.256209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.256188    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d18b6f4-20eb-4162-8202-063ba87f5a43-kube-api-access-4cnqr\" (UniqueName: \"kubernetes.io/projected/0d18b6f4-20eb-4162-8202-063ba87f5a43-kube-api-access-4cnqr\") pod \"0d18b6f4-20eb-4162-8202-063ba87f5a43\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") "
Oct 30 21:11:41.256450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.256425    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-maas-api-tls\") pod \"0d18b6f4-20eb-4162-8202-063ba87f5a43\" (UID: \"0d18b6f4-20eb-4162-8202-063ba87f5a43\") "
Oct 30 21:11:41.256996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.256973    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d18b6f4-20eb-4162-8202-063ba87f5a43-openshift-service-ca" pod "0d18b6f4-20eb-4162-8202-063ba87f5a43" (UID: "0d18b6f4-20eb-4162-8202-063ba87f5a43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:41.261705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.259424    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d18b6f4-20eb-4162-8202-063ba87f5a43-kube-api-access-4cnqr" pod "0d18b6f4-20eb-4162-8202-063ba87f5a43" (UID: "0d18b6f4-20eb-4162-8202-063ba87f5a43"). InnerVolumeSpecName "kube-api-access-4cnqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:41.261705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.261082    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-metrics-certs" pod "0d18b6f4-20eb-4162-8202-063ba87f5a43" (UID: "0d18b6f4-20eb-4162-8202-063ba87f5a43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:41.261705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.261634    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-maas-api-tls" pod "0d18b6f4-20eb-4162-8202-063ba87f5a43" (UID: "0d18b6f4-20eb-4162-8202-063ba87f5a43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:41.358052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.358017    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:41.358052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.358045    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d18b6f4-20eb-4162-8202-063ba87f5a43-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:41.358052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.358054    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d18b6f4-20eb-4162-8202-063ba87f5a43-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:41.358277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.358063    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnqr\" (UniqueName: \"kubernetes.io/projected/0d18b6f4-20eb-4162-8202-063ba87f5a43-kube-api-access-4cnqr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:41.654724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.654695    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"]
Oct 30 21:11:41.685975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.685937    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"]
Oct 30 21:11:41.799996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.799966    4633 generic.go:356] "Generic (PLEG): container finished" podID="0d18b6f4-20eb-4162-8202-063ba87f5a43" containerID="d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283" exitCode=0
Oct 30 21:11:41.801149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.800095    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"
Oct 30 21:11:41.801607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.800114    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8" event={"ID":"0d18b6f4-20eb-4162-8202-063ba87f5a43","Type":"ContainerDied","Data":"d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283"}
Oct 30 21:11:41.801709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.801652    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8" event={"ID":"0d18b6f4-20eb-4162-8202-063ba87f5a43","Type":"ContainerDied","Data":"56047e34231c94064aa459701ff9099a871115800600796e9fbb6a86beb0273f"}
Oct 30 21:11:41.801709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.801705    4633 scope.go:122] "RemoveContainer" containerID="d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283"
Oct 30 21:11:41.805232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.805210    4633 generic.go:356] "Generic (PLEG): container finished" podID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerID="5e22fd1f5d8fa96b0046dd86abfbda9abc2a9bb12cdc56b6d18c3abe737a4626" exitCode=0
Oct 30 21:11:41.805389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.805363    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" event={"ID":"789b6512-9b4b-4504-bd33-630e72d8ba8d","Type":"ContainerDied","Data":"5e22fd1f5d8fa96b0046dd86abfbda9abc2a9bb12cdc56b6d18c3abe737a4626"}
Oct 30 21:11:41.813928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.813906    4633 scope.go:122] "RemoveContainer" containerID="d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283"
Oct 30 21:11:41.814226 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:41.814206    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283\": container with ID starting with d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283 not found: ID does not exist" containerID="d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283"
Oct 30 21:11:41.814286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.814234    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283"} err="failed to get container status \"d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283\": rpc error: code = NotFound desc = could not find container \"d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283\": container with ID starting with d8576601df228a097be5d293dd8063ae5268d85c2b14733e77dd24afd1ef9283 not found: ID does not exist"
Oct 30 21:11:41.814286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.814250    4633 scope.go:122] "RemoveContainer" containerID="48b36fe34c083d6843777295411380ff4035473224733b53af10b3cbe4690b44"
Oct 30 21:11:41.820845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.820819    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"]
Oct 30 21:11:41.824473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.824451    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6eedd82b-b5b65797f-z9ck8"]
Oct 30 21:11:41.926468 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.926440    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965030    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-certs\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965097    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789b6512-9b4b-4504-bd33-630e72d8ba8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789b6512-9b4b-4504-bd33-630e72d8ba8d-istiod-ca-cert\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965130    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-data\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965180    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-socket\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965251    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-kube-api-access-rpvjx\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-kube-api-access-rpvjx\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965319    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-credential-socket\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965358    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-token\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-token\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965410    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-podinfo\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.965446    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-envoy\") pod \"789b6512-9b4b-4504-bd33-630e72d8ba8d\" (UID: \"789b6512-9b4b-4504-bd33-630e72d8ba8d\") "
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.969983    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-credential-socket" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.970196    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-certs" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.970427    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-data" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.970978    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789b6512-9b4b-4504-bd33-630e72d8ba8d-istiod-ca-cert" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:41.971733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:41.971095    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-socket" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:42.002505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.002463    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-podinfo" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:11:42.002800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.002772    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-kube-api-access-rpvjx" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "kube-api-access-rpvjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:42.005920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.005897    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-token" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:42.006424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.006392    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-envoy" pod "789b6512-9b4b-4504-bd33-630e72d8ba8d" (UID: "789b6512-9b4b-4504-bd33-630e72d8ba8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:42.066503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066462    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.066503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066498    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.066776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066514    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.066776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066529    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.066776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066544    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.066776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066558    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/789b6512-9b4b-4504-bd33-630e72d8ba8d-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.066776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066572    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.066776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066586    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/789b6512-9b4b-4504-bd33-630e72d8ba8d-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.066776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.066615    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpvjx\" (UniqueName: \"kubernetes.io/projected/789b6512-9b4b-4504-bd33-630e72d8ba8d-kube-api-access-rpvjx\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:42.537552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.537503    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"]
Oct 30 21:11:42.538796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.538771    4633 cpu_manager.go:414] "containerMap: removing container" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy"
Oct 30 21:11:42.538930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.538918    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy"
Oct 30 21:11:42.539015 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539006    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c413f067-6fe2-4579-8ebe-9524c3f97d27" containerName="payload-processing"
Oct 30 21:11:42.539101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539091    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c413f067-6fe2-4579-8ebe-9524c3f97d27" containerName="payload-processing"
Oct 30 21:11:42.539185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539177    4633 cpu_manager.go:414] "containerMap: removing container" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy"
Oct 30 21:11:42.539247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539238    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy"
Oct 30 21:11:42.539365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539356    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0d18b6f4-20eb-4162-8202-063ba87f5a43" containerName="maas-api"
Oct 30 21:11:42.539442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539433    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d18b6f4-20eb-4162-8202-063ba87f5a43" containerName="maas-api"
Oct 30 21:11:42.539540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539531    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ec68ab05-108c-40d9-8323-bf953c3b7f1f" containerName="payload-pre-processing"
Oct 30 21:11:42.539701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539663    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec68ab05-108c-40d9-8323-bf953c3b7f1f" containerName="payload-pre-processing"
Oct 30 21:11:42.539980 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.539967    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec68ab05-108c-40d9-8323-bf953c3b7f1f" containerName="payload-pre-processing"
Oct 30 21:11:42.540075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.540066    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy"
Oct 30 21:11:42.540158 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.540150    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c413f067-6fe2-4579-8ebe-9524c3f97d27" containerName="payload-processing"
Oct 30 21:11:42.540231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.540223    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" containerName="istio-proxy"
Oct 30 21:11:42.540305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.540297    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d18b6f4-20eb-4162-8202-063ba87f5a43" containerName="maas-api"
Oct 30 21:11:42.550057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.550021    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"]
Oct 30 21:11:42.550057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.550046    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.557362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.557319    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bfc50e-openshift-default-dockercfg-g92dm\""
Oct 30 21:11:42.571978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.571946    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87731e5-07ec-4842-8e0a-f98200c5acf3-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.572121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.572014    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.572121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.572104    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.572233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.572155    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.572233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.572200    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bx6\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-kube-api-access-x9bx6\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.572387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.572363    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.572447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.572410    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.572447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.572440    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.572544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.572479    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.673914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.673876    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.673918    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.673952    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.673981    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674001    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87731e5-07ec-4842-8e0a-f98200c5acf3-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674038    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674064    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674119    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674207    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bx6\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-kube-api-access-x9bx6\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674607    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674613    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674821    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674826    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.674973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.674928    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87731e5-07ec-4842-8e0a-f98200c5acf3-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.677283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.677256    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.678308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.678288    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.693863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.693839    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.695046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.695025    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bx6\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-kube-api-access-x9bx6\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.703965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.703944    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"]
Oct 30 21:11:42.708800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.708783    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:42.720538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.720517    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-05a342\""
Oct 30 21:11:42.723980 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.723905    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-05a342-dockercfg-ztzl8\""
Oct 30 21:11:42.728176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.728156    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"]
Oct 30 21:11:42.742039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.742017    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"]
Oct 30 21:11:42.746301 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.745916    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:42.756647 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.756372    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"]
Oct 30 21:11:42.775176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.775149    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bgwm\" (UniqueName: \"kubernetes.io/projected/75efdde9-a98a-47e0-99cc-9f618bbe57e2-kube-api-access-8bgwm\") pod \"payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s\" (UID: \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:42.775369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.775345    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75efdde9-a98a-47e0-99cc-9f618bbe57e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s\" (UID: \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:42.775430 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.775411    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69stn\" (UniqueName: \"kubernetes.io/projected/6517c30e-d017-4879-a147-c9a10e29f3f0-kube-api-access-69stn\") pod \"payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq\" (UID: \"6517c30e-d017-4879-a147-c9a10e29f3f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:42.775527 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.775503    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6517c30e-d017-4879-a147-c9a10e29f3f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq\" (UID: \"6517c30e-d017-4879-a147-c9a10e29f3f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:42.814151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.814080    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn" event={"ID":"789b6512-9b4b-4504-bd33-630e72d8ba8d","Type":"ContainerDied","Data":"a07e929d9bbbc6744c291a2d6f7b99408e90a12fe05accd086f39ea95fb246be"}
Oct 30 21:11:42.814151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.814111    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"
Oct 30 21:11:42.814151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.814131    4633 scope.go:122] "RemoveContainer" containerID="5e22fd1f5d8fa96b0046dd86abfbda9abc2a9bb12cdc56b6d18c3abe737a4626"
Oct 30 21:11:42.841008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.840986    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"]
Oct 30 21:11:42.846929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.846899    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e239cab2-gw-openshift-default-659cbb77df-wvhrn"]
Oct 30 21:11:42.869821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.869791    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:42.876619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.876583    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bgwm\" (UniqueName: \"kubernetes.io/projected/75efdde9-a98a-47e0-99cc-9f618bbe57e2-kube-api-access-8bgwm\") pod \"payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s\" (UID: \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:42.876830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.876809    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75efdde9-a98a-47e0-99cc-9f618bbe57e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s\" (UID: \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:42.876925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.876880    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69stn\" (UniqueName: \"kubernetes.io/projected/6517c30e-d017-4879-a147-c9a10e29f3f0-kube-api-access-69stn\") pod \"payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq\" (UID: \"6517c30e-d017-4879-a147-c9a10e29f3f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:42.876925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.876914    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6517c30e-d017-4879-a147-c9a10e29f3f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq\" (UID: \"6517c30e-d017-4879-a147-c9a10e29f3f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:42.877889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.877861    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6517c30e-d017-4879-a147-c9a10e29f3f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq\" (UID: \"6517c30e-d017-4879-a147-c9a10e29f3f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:42.878981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.878208    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75efdde9-a98a-47e0-99cc-9f618bbe57e2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s\" (UID: \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:42.891301 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.891277    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69stn\" (UniqueName: \"kubernetes.io/projected/6517c30e-d017-4879-a147-c9a10e29f3f0-kube-api-access-69stn\") pod \"payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq\" (UID: \"6517c30e-d017-4879-a147-c9a10e29f3f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:42.891967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:42.891944    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bgwm\" (UniqueName: \"kubernetes.io/projected/75efdde9-a98a-47e0-99cc-9f618bbe57e2-kube-api-access-8bgwm\") pod \"payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s\" (UID: \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:43.022477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.022442    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:43.068775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.066749    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:43.179736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.178978    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"]
Oct 30 21:11:43.188780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.185885    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"]
Oct 30 21:11:43.188780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.186124    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy" containerID="cri-o://002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f" gracePeriod=30
Oct 30 21:11:43.192568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.192536    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:43.192641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.192623    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:43.192722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.192695    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:43.375295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.375265    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"]
Oct 30 21:11:43.375866 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:43.375838    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6517c30e_d017_4879_a147_c9a10e29f3f0.slice/crio-2faa0c24096ba508a06fc2ec27ea661f2d5c0aa66cbdd63d5111bff2ae7209e9 WatchSource:0}: Error finding container 2faa0c24096ba508a06fc2ec27ea661f2d5c0aa66cbdd63d5111bff2ae7209e9: Status 404 returned error can't find the container with id 2faa0c24096ba508a06fc2ec27ea661f2d5c0aa66cbdd63d5111bff2ae7209e9
Oct 30 21:11:43.410288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.410255    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"]
Oct 30 21:11:43.410579 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:43.410557    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75efdde9_a98a_47e0_99cc_9f618bbe57e2.slice/crio-ac7c50422a87ec90445b522788cbea491830c30809bf22bf8843416c7d3fb878 WatchSource:0}: Error finding container ac7c50422a87ec90445b522788cbea491830c30809bf22bf8843416c7d3fb878: Status 404 returned error can't find the container with id ac7c50422a87ec90445b522788cbea491830c30809bf22bf8843416c7d3fb878
Oct 30 21:11:43.814830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.814799    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d18b6f4-20eb-4162-8202-063ba87f5a43" path="/var/lib/kubelet/pods/0d18b6f4-20eb-4162-8202-063ba87f5a43/volumes"
Oct 30 21:11:43.815339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.815322    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789b6512-9b4b-4504-bd33-630e72d8ba8d" path="/var/lib/kubelet/pods/789b6512-9b4b-4504-bd33-630e72d8ba8d/volumes"
Oct 30 21:11:43.822564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.822540    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq" event={"ID":"6517c30e-d017-4879-a147-c9a10e29f3f0","Type":"ContainerStarted","Data":"448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0"}
Oct 30 21:11:43.822738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.822588    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq" event={"ID":"6517c30e-d017-4879-a147-c9a10e29f3f0","Type":"ContainerStarted","Data":"2faa0c24096ba508a06fc2ec27ea661f2d5c0aa66cbdd63d5111bff2ae7209e9"}
Oct 30 21:11:43.822738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.822684    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:43.825536 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.825481    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" event={"ID":"d87731e5-07ec-4842-8e0a-f98200c5acf3","Type":"ContainerStarted","Data":"47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865"}
Oct 30 21:11:43.825734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.825600    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" event={"ID":"d87731e5-07ec-4842-8e0a-f98200c5acf3","Type":"ContainerStarted","Data":"03c4e79bbfedcce073aabe3adef553ad37f48f4724a2915ab82a4da64d45f561"}
Oct 30 21:11:43.827426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.827400    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s" event={"ID":"75efdde9-a98a-47e0-99cc-9f618bbe57e2","Type":"ContainerStarted","Data":"dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0"}
Oct 30 21:11:43.827509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.827443    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s" event={"ID":"75efdde9-a98a-47e0-99cc-9f618bbe57e2","Type":"ContainerStarted","Data":"ac7c50422a87ec90445b522788cbea491830c30809bf22bf8843416c7d3fb878"}
Oct 30 21:11:43.827561 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.827526    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:43.842270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.842229    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq" podStartSLOduration=1.842213955 podStartE2EDuration="1.842213955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:43.838879995 +0000 UTC m=+612.188607388" watchObservedRunningTime="2026-10-30 21:11:43.842213955 +0000 UTC m=+612.191941352"
Oct 30 21:11:43.864929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.864881    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podStartSLOduration=1.864869428 podStartE2EDuration="1.864869428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:43.863241514 +0000 UTC m=+612.212968910" watchObservedRunningTime="2026-10-30 21:11:43.864869428 +0000 UTC m=+612.214596824"
Oct 30 21:11:43.869908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.869889    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:43.884871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:43.884828    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s" podStartSLOduration=1.884816423 podStartE2EDuration="1.884816423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:43.881445646 +0000 UTC m=+612.231173041" watchObservedRunningTime="2026-10-30 21:11:43.884816423 +0000 UTC m=+612.234543819"
Oct 30 21:11:44.489229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.489196    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"]
Oct 30 21:11:44.493284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.493264    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.498125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.498107    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3011c6-openshift-default-dockercfg-m96xs\""
Oct 30 21:11:44.513316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.513291    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"]
Oct 30 21:11:44.616088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.615557    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.616088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.615602    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae638011-2505-459d-ad37-e3e618d710dc-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.616088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.615638    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.616088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.615707    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.616088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.615752    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.616088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.615825    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.616088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.615872    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.616088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.615902    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79px5\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-kube-api-access-79px5\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.616571 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.616142    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae638011-2505-459d-ad37-e3e618d710dc-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.717534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717496    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae638011-2505-459d-ad37-e3e618d710dc-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.717534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717535    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.717802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717560    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae638011-2505-459d-ad37-e3e618d710dc-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.717802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717609    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.717802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717646    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.717802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717706    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.717802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717768    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.718039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717819    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.718039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717854    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79px5\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-kube-api-access-79px5\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.718039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.717982    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.718322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.718296    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.718322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.718314    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.718463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.718382    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae638011-2505-459d-ad37-e3e618d710dc-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.718463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.718384    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.720620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.720597    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.720738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.720721    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae638011-2505-459d-ad37-e3e618d710dc-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.732330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.732297    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.738097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.738075    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79px5\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-kube-api-access-79px5\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.754861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.754746    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9" podUID="1a69c553-0696-4fd3-aa8a-e3c722811e1f" containerName="payload-pre-processing" containerID="cri-o://1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a" gracePeriod=25
Oct 30 21:11:44.808316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.808285    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:44.882525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:44.881896    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:45.072411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.070049    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:45.114572 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:45.114515    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae638011_2505_459d_ad37_e3e618d710dc.slice/crio-edda11b70510327a3b182d85e3d35b395cd16eae9274c5222b0239bf2d967f21 WatchSource:0}: Error finding container edda11b70510327a3b182d85e3d35b395cd16eae9274c5222b0239bf2d967f21: Status 404 returned error can't find the container with id edda11b70510327a3b182d85e3d35b395cd16eae9274c5222b0239bf2d967f21
Oct 30 21:11:45.119219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.116925    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:45.119219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.117017    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:45.119219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.117061    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:11:45.124743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.120664    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"]
Oct 30 21:11:45.124743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.121712    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a69c553-0696-4fd3-aa8a-e3c722811e1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69c553-0696-4fd3-aa8a-e3c722811e1f-plugins-config-volume\") pod \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\" (UID: \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\") "
Oct 30 21:11:45.124743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.121975    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a69c553-0696-4fd3-aa8a-e3c722811e1f-kube-api-access-zlvqs\" (UniqueName: \"kubernetes.io/projected/1a69c553-0696-4fd3-aa8a-e3c722811e1f-kube-api-access-zlvqs\") pod \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\" (UID: \"1a69c553-0696-4fd3-aa8a-e3c722811e1f\") "
Oct 30 21:11:45.133788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.133737    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a69c553-0696-4fd3-aa8a-e3c722811e1f-plugins-config-volume" pod "1a69c553-0696-4fd3-aa8a-e3c722811e1f" (UID: "1a69c553-0696-4fd3-aa8a-e3c722811e1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:45.145902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.145861    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a69c553-0696-4fd3-aa8a-e3c722811e1f-kube-api-access-zlvqs" pod "1a69c553-0696-4fd3-aa8a-e3c722811e1f" (UID: "1a69c553-0696-4fd3-aa8a-e3c722811e1f"). InnerVolumeSpecName "kube-api-access-zlvqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:45.223648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.223621    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvqs\" (UniqueName: \"kubernetes.io/projected/1a69c553-0696-4fd3-aa8a-e3c722811e1f-kube-api-access-zlvqs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:45.223648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.223649    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a69c553-0696-4fd3-aa8a-e3c722811e1f-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:45.845014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.844979    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" event={"ID":"ae638011-2505-459d-ad37-e3e618d710dc","Type":"ContainerStarted","Data":"18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43"}
Oct 30 21:11:45.845489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.845034    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" event={"ID":"ae638011-2505-459d-ad37-e3e618d710dc","Type":"ContainerStarted","Data":"edda11b70510327a3b182d85e3d35b395cd16eae9274c5222b0239bf2d967f21"}
Oct 30 21:11:45.846349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.846326    4633 generic.go:356] "Generic (PLEG): container finished" podID="1a69c553-0696-4fd3-aa8a-e3c722811e1f" containerID="1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a" exitCode=0
Oct 30 21:11:45.846471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.846365    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"
Oct 30 21:11:45.846471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.846379    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9" event={"ID":"1a69c553-0696-4fd3-aa8a-e3c722811e1f","Type":"ContainerDied","Data":"1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a"}
Oct 30 21:11:45.846471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.846419    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9" event={"ID":"1a69c553-0696-4fd3-aa8a-e3c722811e1f","Type":"ContainerDied","Data":"751e8b13d5de5a056f24d4a5e5674e7393dacce73c3642e9d59fc9f9e33b36c2"}
Oct 30 21:11:45.846471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.846443    4633 scope.go:122] "RemoveContainer" containerID="1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a"
Oct 30 21:11:45.846812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.846784    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:45.847997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.847978    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:11:45.855984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.855966    4633 scope.go:122] "RemoveContainer" containerID="1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a"
Oct 30 21:11:45.856291 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:45.856271    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a\": container with ID starting with 1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a not found: ID does not exist" containerID="1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a"
Oct 30 21:11:45.856350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.856300    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a"} err="failed to get container status \"1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a\": rpc error: code = NotFound desc = could not find container \"1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a\": container with ID starting with 1e14d425368db68c74b8a02668536ca60fa0102a02c8c1bb5e311fbab274ff2a not found: ID does not exist"
Oct 30 21:11:45.871522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.867967    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" podStartSLOduration=1.86795167 podStartE2EDuration="1.86795167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:45.863546608 +0000 UTC m=+614.213274006" watchObservedRunningTime="2026-10-30 21:11:45.86795167 +0000 UTC m=+614.217679068"
Oct 30 21:11:45.900354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.900312    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"]
Oct 30 21:11:45.903285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:45.903253    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6eedd82b-cfc45487f-fl6b9"]
Oct 30 21:11:46.662447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.660752    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq" podUID="17fff94e-995e-4eee-bc7a-67d66ed68b50" containerName="payload-processing" containerID="cri-o://88bab4e7bc3cc4e0ffdcacf883f6d28546971ab2db5cb67b0a6df3e1bc30a423" gracePeriod=25
Oct 30 21:11:46.687032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.686748    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp" podUID="f2a9d602-2259-47e3-bf24-de836933255a" containerName="payload-pre-processing" containerID="cri-o://b8c027fa7278aa260f91e8147b92b147f7b83314d403e7557a936ddf9a828fa2" gracePeriod=25
Oct 30 21:11:46.782234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.782195    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:11:46.811283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.808604    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:46.817175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.817000    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:46.864453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.864258    4633 generic.go:356] "Generic (PLEG): container finished" podID="17fff94e-995e-4eee-bc7a-67d66ed68b50" containerID="88bab4e7bc3cc4e0ffdcacf883f6d28546971ab2db5cb67b0a6df3e1bc30a423" exitCode=0
Oct 30 21:11:46.864453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.864355    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq" event={"ID":"17fff94e-995e-4eee-bc7a-67d66ed68b50","Type":"ContainerDied","Data":"88bab4e7bc3cc4e0ffdcacf883f6d28546971ab2db5cb67b0a6df3e1bc30a423"}
Oct 30 21:11:46.868691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.868337    4633 generic.go:356] "Generic (PLEG): container finished" podID="f2a9d602-2259-47e3-bf24-de836933255a" containerID="b8c027fa7278aa260f91e8147b92b147f7b83314d403e7557a936ddf9a828fa2" exitCode=0
Oct 30 21:11:46.870356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.869010    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp" event={"ID":"f2a9d602-2259-47e3-bf24-de836933255a","Type":"ContainerDied","Data":"b8c027fa7278aa260f91e8147b92b147f7b83314d403e7557a936ddf9a828fa2"}
Oct 30 21:11:46.870356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.869431    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:46.871828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.871536    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:11:46.996135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.996106    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"]
Oct 30 21:11:46.996552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:46.996526    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz" podUID="12a20f96-68c1-440d-ae36-653d4afa8684" containerName="maas-api" containerID="cri-o://bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad" gracePeriod=30
Oct 30 21:11:47.071961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.068859    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:47.074235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.074209    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:47.156630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.155187    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a9d602-2259-47e3-bf24-de836933255a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a9d602-2259-47e3-bf24-de836933255a-plugins-config-volume\") pod \"f2a9d602-2259-47e3-bf24-de836933255a\" (UID: \"f2a9d602-2259-47e3-bf24-de836933255a\") "
Oct 30 21:11:47.156630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.155296    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fff94e-995e-4eee-bc7a-67d66ed68b50-kube-api-access-jt4vq\" (UniqueName: \"kubernetes.io/projected/17fff94e-995e-4eee-bc7a-67d66ed68b50-kube-api-access-jt4vq\") pod \"17fff94e-995e-4eee-bc7a-67d66ed68b50\" (UID: \"17fff94e-995e-4eee-bc7a-67d66ed68b50\") "
Oct 30 21:11:47.156630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.155335    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fff94e-995e-4eee-bc7a-67d66ed68b50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fff94e-995e-4eee-bc7a-67d66ed68b50-plugins-config-volume\") pod \"17fff94e-995e-4eee-bc7a-67d66ed68b50\" (UID: \"17fff94e-995e-4eee-bc7a-67d66ed68b50\") "
Oct 30 21:11:47.156630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.155392    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a9d602-2259-47e3-bf24-de836933255a-kube-api-access-qwxjf\" (UniqueName: \"kubernetes.io/projected/f2a9d602-2259-47e3-bf24-de836933255a-kube-api-access-qwxjf\") pod \"f2a9d602-2259-47e3-bf24-de836933255a\" (UID: \"f2a9d602-2259-47e3-bf24-de836933255a\") "
Oct 30 21:11:47.179794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.166222    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a9d602-2259-47e3-bf24-de836933255a-kube-api-access-qwxjf" pod "f2a9d602-2259-47e3-bf24-de836933255a" (UID: "f2a9d602-2259-47e3-bf24-de836933255a"). InnerVolumeSpecName "kube-api-access-qwxjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:47.179794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.177876    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fff94e-995e-4eee-bc7a-67d66ed68b50-plugins-config-volume" pod "17fff94e-995e-4eee-bc7a-67d66ed68b50" (UID: "17fff94e-995e-4eee-bc7a-67d66ed68b50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:47.184696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.181041    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a9d602-2259-47e3-bf24-de836933255a-plugins-config-volume" pod "f2a9d602-2259-47e3-bf24-de836933255a" (UID: "f2a9d602-2259-47e3-bf24-de836933255a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:47.185292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.185254    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fff94e-995e-4eee-bc7a-67d66ed68b50-kube-api-access-jt4vq" pod "17fff94e-995e-4eee-bc7a-67d66ed68b50" (UID: "17fff94e-995e-4eee-bc7a-67d66ed68b50"). InnerVolumeSpecName "kube-api-access-jt4vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:47.285770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.261405    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a9d602-2259-47e3-bf24-de836933255a-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:47.285770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.261444    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt4vq\" (UniqueName: \"kubernetes.io/projected/17fff94e-995e-4eee-bc7a-67d66ed68b50-kube-api-access-jt4vq\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:47.285770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.261462    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17fff94e-995e-4eee-bc7a-67d66ed68b50-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:47.285770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.261475    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwxjf\" (UniqueName: \"kubernetes.io/projected/f2a9d602-2259-47e3-bf24-de836933255a-kube-api-access-qwxjf\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:47.442496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.442469    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:47.564557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.564464    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls\") pod \"12a20f96-68c1-440d-ae36-653d4afa8684\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") "
Oct 30 21:11:47.564744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.564569    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a20f96-68c1-440d-ae36-653d4afa8684-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a20f96-68c1-440d-ae36-653d4afa8684-openshift-service-ca\") pod \"12a20f96-68c1-440d-ae36-653d4afa8684\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") "
Oct 30 21:11:47.564744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.564618    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-metrics-certs\") pod \"12a20f96-68c1-440d-ae36-653d4afa8684\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") "
Oct 30 21:11:47.564744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.564649    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a20f96-68c1-440d-ae36-653d4afa8684-kube-api-access-9jlm2\" (UniqueName: \"kubernetes.io/projected/12a20f96-68c1-440d-ae36-653d4afa8684-kube-api-access-9jlm2\") pod \"12a20f96-68c1-440d-ae36-653d4afa8684\" (UID: \"12a20f96-68c1-440d-ae36-653d4afa8684\") "
Oct 30 21:11:47.565096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.565058    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a20f96-68c1-440d-ae36-653d4afa8684-openshift-service-ca" pod "12a20f96-68c1-440d-ae36-653d4afa8684" (UID: "12a20f96-68c1-440d-ae36-653d4afa8684"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:47.566952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.566926    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls" pod "12a20f96-68c1-440d-ae36-653d4afa8684" (UID: "12a20f96-68c1-440d-ae36-653d4afa8684"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:47.567067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.566979    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-metrics-certs" pod "12a20f96-68c1-440d-ae36-653d4afa8684" (UID: "12a20f96-68c1-440d-ae36-653d4afa8684"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:47.567067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.566987    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a20f96-68c1-440d-ae36-653d4afa8684-kube-api-access-9jlm2" pod "12a20f96-68c1-440d-ae36-653d4afa8684" (UID: "12a20f96-68c1-440d-ae36-653d4afa8684"). InnerVolumeSpecName "kube-api-access-9jlm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:47.666464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.666423    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:47.666464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.666463    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jlm2\" (UniqueName: \"kubernetes.io/projected/12a20f96-68c1-440d-ae36-653d4afa8684-kube-api-access-9jlm2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:47.666775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.666479    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a20f96-68c1-440d-ae36-653d4afa8684-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:47.666775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.666493    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a20f96-68c1-440d-ae36-653d4afa8684-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:47.799493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.799443    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a69c553-0696-4fd3-aa8a-e3c722811e1f" path="/var/lib/kubelet/pods/1a69c553-0696-4fd3-aa8a-e3c722811e1f/volumes"
Oct 30 21:11:47.877326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.877238    4633 generic.go:356] "Generic (PLEG): container finished" podID="12a20f96-68c1-440d-ae36-653d4afa8684" containerID="bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad" exitCode=0
Oct 30 21:11:47.877326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.877293    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"
Oct 30 21:11:47.877923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.877324    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz" event={"ID":"12a20f96-68c1-440d-ae36-653d4afa8684","Type":"ContainerDied","Data":"bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad"}
Oct 30 21:11:47.877923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.877372    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz" event={"ID":"12a20f96-68c1-440d-ae36-653d4afa8684","Type":"ContainerDied","Data":"a68540afd1e89948f56474957369f606e332d6ce91056f1a4f1e8640dd3b5a31"}
Oct 30 21:11:47.877923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.877400    4633 scope.go:122] "RemoveContainer" containerID="bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad"
Oct 30 21:11:47.879227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.879203    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"
Oct 30 21:11:47.879338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.879243    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq" event={"ID":"17fff94e-995e-4eee-bc7a-67d66ed68b50","Type":"ContainerDied","Data":"2497f139571a354a04ad0e1bfe040837ab4aa203cdf17e5c74c08652291a7485"}
Oct 30 21:11:47.881253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.881198    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp" event={"ID":"f2a9d602-2259-47e3-bf24-de836933255a","Type":"ContainerDied","Data":"1dd383544abc82c356295e627088253ebe9ff29dba6545e2dbc3e2999f744832"}
Oct 30 21:11:47.881253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.881214    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"
Oct 30 21:11:47.889375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.889358    4633 scope.go:122] "RemoveContainer" containerID="bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad"
Oct 30 21:11:47.889620 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:47.889599    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad\": container with ID starting with bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad not found: ID does not exist" containerID="bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad"
Oct 30 21:11:47.889724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.889631    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad"} err="failed to get container status \"bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad\": rpc error: code = NotFound desc = could not find container \"bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad\": container with ID starting with bccf0016d6eaba78124254030d296979d4591e39e321a26b42b5a1efcaa35bad not found: ID does not exist"
Oct 30 21:11:47.889724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.889653    4633 scope.go:122] "RemoveContainer" containerID="88bab4e7bc3cc4e0ffdcacf883f6d28546971ab2db5cb67b0a6df3e1bc30a423"
Oct 30 21:11:47.898587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.898572    4633 scope.go:122] "RemoveContainer" containerID="b8c027fa7278aa260f91e8147b92b147f7b83314d403e7557a936ddf9a828fa2"
Oct 30 21:11:47.998415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:47.998381    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"]
Oct 30 21:11:48.002585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.002558    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-111fd691-69c7b964d7-6wqsp"]
Oct 30 21:11:48.024699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.015851    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"]
Oct 30 21:11:48.055369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.055341    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-111fd691-cc48f764f-8tvvq"]
Oct 30 21:11:48.089786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.086740    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"]
Oct 30 21:11:48.092200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.092169    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5ee9aa6f-5fc86ffbb4-92bwz"]
Oct 30 21:11:48.550887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.550861    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:48.689968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.689936    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"]
Oct 30 21:11:48.690478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690459    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f2a9d602-2259-47e3-bf24-de836933255a" containerName="payload-pre-processing"
Oct 30 21:11:48.690478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690478    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a9d602-2259-47e3-bf24-de836933255a" containerName="payload-pre-processing"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690488    4633 cpu_manager.go:414] "containerMap: removing container" podUID="17fff94e-995e-4eee-bc7a-67d66ed68b50" containerName="payload-processing"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690494    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fff94e-995e-4eee-bc7a-67d66ed68b50" containerName="payload-processing"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690503    4633 cpu_manager.go:414] "containerMap: removing container" podUID="12a20f96-68c1-440d-ae36-653d4afa8684" containerName="maas-api"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690510    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a20f96-68c1-440d-ae36-653d4afa8684" containerName="maas-api"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690540    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690545    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690553    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690557    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690568    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1a69c553-0696-4fd3-aa8a-e3c722811e1f" containerName="payload-pre-processing"
Oct 30 21:11:48.690625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690572    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a69c553-0696-4fd3-aa8a-e3c722811e1f" containerName="payload-pre-processing"
Oct 30 21:11:48.691144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690647    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy"
Oct 30 21:11:48.691144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690655    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy"
Oct 30 21:11:48.691144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690661    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a69c553-0696-4fd3-aa8a-e3c722811e1f" containerName="payload-pre-processing"
Oct 30 21:11:48.691144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690685    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a9d602-2259-47e3-bf24-de836933255a" containerName="payload-pre-processing"
Oct 30 21:11:48.691144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690694    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fff94e-995e-4eee-bc7a-67d66ed68b50" containerName="payload-processing"
Oct 30 21:11:48.691144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.690700    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a20f96-68c1-440d-ae36-653d4afa8684" containerName="maas-api"
Oct 30 21:11:48.693998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.693976    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.694133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694106    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-certs\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.694204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694158    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-socket\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.694261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694222    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-istio-token\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-istio-token\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.694313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694283    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9516543-6c63-449d-8618-72df184b0a2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9516543-6c63-449d-8618-72df184b0a2f-istiod-ca-cert\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.694387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694371    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9516543-6c63-449d-8618-72df184b0a2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9516543-6c63-449d-8618-72df184b0a2f-istio-podinfo\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.694499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694458    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-envoy\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.694572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694553    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-data\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.694626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694614    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-credential-socket\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.694696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.694656    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-kube-api-access-4w82f\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-kube-api-access-4w82f\") pod \"d9516543-6c63-449d-8618-72df184b0a2f\" (UID: \"d9516543-6c63-449d-8618-72df184b0a2f\") "
Oct 30 21:11:48.696074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.695455    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-data" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:48.696074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.695772    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9516543-6c63-449d-8618-72df184b0a2f-istiod-ca-cert" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:48.696074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.695914    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-certs" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:48.696074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.696045    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-credential-socket" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:48.696372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.696163    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-socket" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:48.701012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.700969    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-envoy" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:48.701012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.700999    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-kube-api-access-4w82f" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "kube-api-access-4w82f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:48.701279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.701251    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-istio-token" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:48.704950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.701875    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-bfc50e-serving-cert\""
Oct 30 21:11:48.705336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.705313    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9516543-6c63-449d-8618-72df184b0a2f-istio-podinfo" pod "d9516543-6c63-449d-8618-72df184b0a2f" (UID: "d9516543-6c63-449d-8618-72df184b0a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:11:48.744690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.731374    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:11:48.744690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.731477    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:11:48.744690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.731564    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rx79k\""
Oct 30 21:11:48.744690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.731662    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:11:48.744690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.737788    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"]
Oct 30 21:11:48.796255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796175    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.796255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796242    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8plqt\" (UniqueName: \"kubernetes.io/projected/f4238ff6-1550-434d-979b-f44b9bd50a99-kube-api-access-8plqt\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.796437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796336    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-metrics-certs\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.796437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796385    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4238ff6-1550-434d-979b-f44b9bd50a99-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.796545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796526    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.796588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796549    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.796588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796560    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w82f\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-kube-api-access-4w82f\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.796588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796571    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.796588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796581    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.796740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796590    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9516543-6c63-449d-8618-72df184b0a2f-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.796740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796599    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9516543-6c63-449d-8618-72df184b0a2f-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.796740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796607    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9516543-6c63-449d-8618-72df184b0a2f-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.796740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.796617    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9516543-6c63-449d-8618-72df184b0a2f-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:48.889454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.889418    4633 generic.go:356] "Generic (PLEG): container finished" podID="d9516543-6c63-449d-8618-72df184b0a2f" containerID="002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f" exitCode=0
Oct 30 21:11:48.889959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.889471    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"
Oct 30 21:11:48.889959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.889500    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" event={"ID":"d9516543-6c63-449d-8618-72df184b0a2f","Type":"ContainerDied","Data":"002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f"}
Oct 30 21:11:48.889959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.889560    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" event={"ID":"d9516543-6c63-449d-8618-72df184b0a2f","Type":"ContainerDied","Data":"8f28de7aa10594fc1f6d9405928b5396eb4843d13dc8722686bbd1c91a67f7c7"}
Oct 30 21:11:48.889959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.889582    4633 scope.go:122] "RemoveContainer" containerID="002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f"
Oct 30 21:11:48.897879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.897854    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8plqt\" (UniqueName: \"kubernetes.io/projected/f4238ff6-1550-434d-979b-f44b9bd50a99-kube-api-access-8plqt\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.897999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.897898    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-metrics-certs\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.897999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.897935    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4238ff6-1550-434d-979b-f44b9bd50a99-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.898218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.898023    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.899071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.899045    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4238ff6-1550-434d-979b-f44b9bd50a99-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.900575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.900554    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-maas-api-tls\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.900828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.900809    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-metrics-certs\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.901689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.901651    4633 scope.go:122] "RemoveContainer" containerID="ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58"
Oct 30 21:11:48.911129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.911111    4633 scope.go:122] "RemoveContainer" containerID="002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f"
Oct 30 21:11:48.911378 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:48.911357    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f\": container with ID starting with 002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f not found: ID does not exist" containerID="002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f"
Oct 30 21:11:48.911427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.911384    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f"} err="failed to get container status \"002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f\": rpc error: code = NotFound desc = could not find container \"002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f\": container with ID starting with 002a2ee17b750d42133fa220a8d08453e9ec33424249a9af5bba806b5b61357f not found: ID does not exist"
Oct 30 21:11:48.911427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.911402    4633 scope.go:122] "RemoveContainer" containerID="ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58"
Oct 30 21:11:48.911619 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:48.911603    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58\": container with ID starting with ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58 not found: ID does not exist" containerID="ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58"
Oct 30 21:11:48.911684 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.911623    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58"} err="failed to get container status \"ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58\": rpc error: code = NotFound desc = could not find container \"ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58\": container with ID starting with ea41b77b9eb4fff75b8336dc3caa30f4d43e979d6304e7d764d1348344436a58 not found: ID does not exist"
Oct 30 21:11:48.939380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.939343    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8plqt\" (UniqueName: \"kubernetes.io/projected/f4238ff6-1550-434d-979b-f44b9bd50a99-kube-api-access-8plqt\") pod \"maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:48.941611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.940820    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"]
Oct 30 21:11:48.944768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:48.944743    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc"]
Oct 30 21:11:49.062895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.062812    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:49.402983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.402956    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"]
Oct 30 21:11:49.404982 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:49.404952    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4238ff6_1550_434d_979b_f44b9bd50a99.slice/crio-d7c2215cf0f86c3c578b49a633b8b7d72bee82905b9876874f11f840ba138e86 WatchSource:0}: Error finding container d7c2215cf0f86c3c578b49a633b8b7d72bee82905b9876874f11f840ba138e86: Status 404 returned error can't find the container with id d7c2215cf0f86c3c578b49a633b8b7d72bee82905b9876874f11f840ba138e86
Oct 30 21:11:49.406564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.406548    4633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:11:49.464312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.464276    4633 patch_prober.go:29] interesting pod/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:11:49.464526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.464347    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6eedd82b-openshift-default-74d5bb4f65-9xkpc" podUID="d9516543-6c63-449d-8618-72df184b0a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:11:49.702115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.702033    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"]
Oct 30 21:11:49.706966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.706945    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:49.745357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.745081    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3011c6\""
Oct 30 21:11:49.750656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.750629    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"]
Oct 30 21:11:49.752115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.752080    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3011c6-dockercfg-bwcf8\""
Oct 30 21:11:49.753950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.753926    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"]
Oct 30 21:11:49.757725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.757705    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"]
Oct 30 21:11:49.757855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.757820    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:49.789986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.789953    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a20f96-68c1-440d-ae36-653d4afa8684" path="/var/lib/kubelet/pods/12a20f96-68c1-440d-ae36-653d4afa8684/volumes"
Oct 30 21:11:49.790489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.790466    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fff94e-995e-4eee-bc7a-67d66ed68b50" path="/var/lib/kubelet/pods/17fff94e-995e-4eee-bc7a-67d66ed68b50/volumes"
Oct 30 21:11:49.791033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.791011    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9516543-6c63-449d-8618-72df184b0a2f" path="/var/lib/kubelet/pods/d9516543-6c63-449d-8618-72df184b0a2f/volumes"
Oct 30 21:11:49.791775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.791747    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a9d602-2259-47e3-bf24-de836933255a" path="/var/lib/kubelet/pods/f2a9d602-2259-47e3-bf24-de836933255a/volumes"
Oct 30 21:11:49.849385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.849350    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef2c07d-1cb3-4b21-a25f-a525adb9a011-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf\" (UID: \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:49.849539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.849416    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/e482d2d9-f8db-4d89-b704-73aa74c6417c-kube-api-access-zswm2\") pod \"payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq\" (UID: \"e482d2d9-f8db-4d89-b704-73aa74c6417c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:49.849581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.849555    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lj4\" (UniqueName: \"kubernetes.io/projected/fef2c07d-1cb3-4b21-a25f-a525adb9a011-kube-api-access-m6lj4\") pod \"payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf\" (UID: \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:49.849622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.849600    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e482d2d9-f8db-4d89-b704-73aa74c6417c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq\" (UID: \"e482d2d9-f8db-4d89-b704-73aa74c6417c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:49.951734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.951659    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lj4\" (UniqueName: \"kubernetes.io/projected/fef2c07d-1cb3-4b21-a25f-a525adb9a011-kube-api-access-m6lj4\") pod \"payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf\" (UID: \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:49.952344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.952280    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e482d2d9-f8db-4d89-b704-73aa74c6417c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq\" (UID: \"e482d2d9-f8db-4d89-b704-73aa74c6417c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:49.955848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.955818    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef2c07d-1cb3-4b21-a25f-a525adb9a011-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf\" (UID: \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:49.956176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.956153    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/e482d2d9-f8db-4d89-b704-73aa74c6417c-kube-api-access-zswm2\") pod \"payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq\" (UID: \"e482d2d9-f8db-4d89-b704-73aa74c6417c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:49.956464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.956434    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e482d2d9-f8db-4d89-b704-73aa74c6417c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq\" (UID: \"e482d2d9-f8db-4d89-b704-73aa74c6417c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:49.956618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.956594    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef2c07d-1cb3-4b21-a25f-a525adb9a011-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf\" (UID: \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:49.965132 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.965071    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp" event={"ID":"f4238ff6-1550-434d-979b-f44b9bd50a99","Type":"ContainerStarted","Data":"d7c2215cf0f86c3c578b49a633b8b7d72bee82905b9876874f11f840ba138e86"}
Oct 30 21:11:49.975758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.975735    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lj4\" (UniqueName: \"kubernetes.io/projected/fef2c07d-1cb3-4b21-a25f-a525adb9a011-kube-api-access-m6lj4\") pod \"payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf\" (UID: \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:49.975862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:49.975823    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/e482d2d9-f8db-4d89-b704-73aa74c6417c-kube-api-access-zswm2\") pod \"payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq\" (UID: \"e482d2d9-f8db-4d89-b704-73aa74c6417c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:50.055997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.055969    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:50.074658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.074630    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:50.375871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.375788    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"]
Oct 30 21:11:50.441150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.421189    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"]
Oct 30 21:11:50.692710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.692659    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"]
Oct 30 21:11:50.696627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.696609    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:50.701848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.701831    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-384164-dockercfg-lwrz9\""
Oct 30 21:11:50.703466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.703449    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-384164\""
Oct 30 21:11:50.742640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.742550    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"]
Oct 30 21:11:50.868464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.868423    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f050773-b095-4cae-b517-d994151796d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn\" (UID: \"4f050773-b095-4cae-b517-d994151796d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:50.868650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.868484    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xcj\" (UniqueName: \"kubernetes.io/projected/4f050773-b095-4cae-b517-d994151796d3-kube-api-access-27xcj\") pod \"payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn\" (UID: \"4f050773-b095-4cae-b517-d994151796d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:50.975983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.975938    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f050773-b095-4cae-b517-d994151796d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn\" (UID: \"4f050773-b095-4cae-b517-d994151796d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:50.976369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.976024    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xcj\" (UniqueName: \"kubernetes.io/projected/4f050773-b095-4cae-b517-d994151796d3-kube-api-access-27xcj\") pod \"payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn\" (UID: \"4f050773-b095-4cae-b517-d994151796d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:50.978584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.978556    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f050773-b095-4cae-b517-d994151796d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn\" (UID: \"4f050773-b095-4cae-b517-d994151796d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:50.984411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.984381    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq" event={"ID":"e482d2d9-f8db-4d89-b704-73aa74c6417c","Type":"ContainerStarted","Data":"0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b"}
Oct 30 21:11:50.984570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.984432    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq" event={"ID":"e482d2d9-f8db-4d89-b704-73aa74c6417c","Type":"ContainerStarted","Data":"437336a92e7816196ab39dfba0fa31daf5a3064964a0b135876966f05ec4ec02"}
Oct 30 21:11:50.985138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.985116    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:11:50.987863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.987833    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp" event={"ID":"f4238ff6-1550-434d-979b-f44b9bd50a99","Type":"ContainerStarted","Data":"b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162"}
Oct 30 21:11:50.988177 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.988114    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:50.990584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.990558    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xcj\" (UniqueName: \"kubernetes.io/projected/4f050773-b095-4cae-b517-d994151796d3-kube-api-access-27xcj\") pod \"payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn\" (UID: \"4f050773-b095-4cae-b517-d994151796d3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:50.991854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.991826    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf" event={"ID":"fef2c07d-1cb3-4b21-a25f-a525adb9a011","Type":"ContainerStarted","Data":"c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c"}
Oct 30 21:11:50.991958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.991876    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf" event={"ID":"fef2c07d-1cb3-4b21-a25f-a525adb9a011","Type":"ContainerStarted","Data":"3df67936207ee7c896f606a643cf52a23f8954e9746284dfdb9afd603ad905fc"}
Oct 30 21:11:50.992443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:50.992421    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:11:51.010129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.009836    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:51.044623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.043021    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq" podStartSLOduration=2.042996829 podStartE2EDuration="2.042996829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:51.012408924 +0000 UTC m=+619.362136321" watchObservedRunningTime="2026-10-30 21:11:51.042996829 +0000 UTC m=+619.392724227"
Oct 30 21:11:51.049206 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.049146    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp" podStartSLOduration=2.501484698 podStartE2EDuration="3.0491279s" totalImagesPullingTime="547.643202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:51.039116375 +0000 UTC m=+619.388843772" watchObservedRunningTime="2026-10-30 21:11:51.0491279 +0000 UTC m=+619.398855297"
Oct 30 21:11:51.066231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.066185    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf" podStartSLOduration=2.066171423 podStartE2EDuration="2.066171423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:51.063323704 +0000 UTC m=+619.413051100" watchObservedRunningTime="2026-10-30 21:11:51.066171423 +0000 UTC m=+619.415898818"
Oct 30 21:11:51.365824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.365793    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"]
Oct 30 21:11:51.366063 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:51.365813    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f050773_b095_4cae_b517_d994151796d3.slice/crio-0508eee384a11a3a770da1532f3d3eff4b1971a8d4dfd94000a59a9f4a608f01 WatchSource:0}: Error finding container 0508eee384a11a3a770da1532f3d3eff4b1971a8d4dfd94000a59a9f4a608f01: Status 404 returned error can't find the container with id 0508eee384a11a3a770da1532f3d3eff4b1971a8d4dfd94000a59a9f4a608f01
Oct 30 21:11:51.966406 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.966369    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"]
Oct 30 21:11:51.971257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.971231    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:51.972973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.972934    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"]
Oct 30 21:11:51.976774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.976743    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-384164-serving-cert\""
Oct 30 21:11:51.993778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.993747    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gwh\" (UniqueName: \"kubernetes.io/projected/1267f93a-e77d-4c6f-b3c0-eef84202ca93-kube-api-access-l8gwh\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:51.993941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.993814    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-metrics-certs\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:51.993941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.993854    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:51.993941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:51.993888    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1267f93a-e77d-4c6f-b3c0-eef84202ca93-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.009143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.009107    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" event={"ID":"4f050773-b095-4cae-b517-d994151796d3","Type":"ContainerStarted","Data":"21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86"}
Oct 30 21:11:52.009288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.009189    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:11:52.009288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.009216    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" event={"ID":"4f050773-b095-4cae-b517-d994151796d3","Type":"ContainerStarted","Data":"0508eee384a11a3a770da1532f3d3eff4b1971a8d4dfd94000a59a9f4a608f01"}
Oct 30 21:11:52.072696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.042549    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" podStartSLOduration=2.042526517 podStartE2EDuration="2.042526517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:52.030984006 +0000 UTC m=+620.380711418" watchObservedRunningTime="2026-10-30 21:11:52.042526517 +0000 UTC m=+620.392253916"
Oct 30 21:11:52.094987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.094942    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gwh\" (UniqueName: \"kubernetes.io/projected/1267f93a-e77d-4c6f-b3c0-eef84202ca93-kube-api-access-l8gwh\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.095142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.095054    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-metrics-certs\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.095187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.095143    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.095222 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.095182    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1267f93a-e77d-4c6f-b3c0-eef84202ca93-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.097822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.097796    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1267f93a-e77d-4c6f-b3c0-eef84202ca93-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.097955 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:52.097911    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-serving-cert: secret "maas-api-e2e-worker-w0-384164-serving-cert" not found
Oct 30 21:11:52.098025 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:52.097982    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls podName:1267f93a-e77d-4c6f-b3c0-eef84202ca93 nodeName:}" failed. No retries permitted until 2026-10-30 21:11:52.597963096 +0000 UTC m=+620.947690476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls") pod "maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb" (UID: "1267f93a-e77d-4c6f-b3c0-eef84202ca93") : secret "maas-api-e2e-worker-w0-384164-serving-cert" not found
Oct 30 21:11:52.103451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.103427    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-metrics-certs\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.172784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.172388    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gwh\" (UniqueName: \"kubernetes.io/projected/1267f93a-e77d-4c6f-b3c0-eef84202ca93-kube-api-access-l8gwh\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.600728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.600658    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.603499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.603472    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls\") pod \"maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:52.609406 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:52.609384    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:53.075805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:53.075296    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"]
Oct 30 21:11:53.078520 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:11:53.078493    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1267f93a_e77d_4c6f_b3c0_eef84202ca93.slice/crio-480a494b9b9c09091f0e8836c67a9a7aad1cf07ba3a40ff26e8a4a5c7002c2a7 WatchSource:0}: Error finding container 480a494b9b9c09091f0e8836c67a9a7aad1cf07ba3a40ff26e8a4a5c7002c2a7: Status 404 returned error can't find the container with id 480a494b9b9c09091f0e8836c67a9a7aad1cf07ba3a40ff26e8a4a5c7002c2a7
Oct 30 21:11:54.074482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:54.074445    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb" event={"ID":"1267f93a-e77d-4c6f-b3c0-eef84202ca93","Type":"ContainerStarted","Data":"c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149"}
Oct 30 21:11:54.074689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:54.074506    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb" event={"ID":"1267f93a-e77d-4c6f-b3c0-eef84202ca93","Type":"ContainerStarted","Data":"480a494b9b9c09091f0e8836c67a9a7aad1cf07ba3a40ff26e8a4a5c7002c2a7"}
Oct 30 21:11:54.074689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:54.074592    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:11:54.095112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:54.095050    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb" podStartSLOduration=2.651292275 podStartE2EDuration="3.095028719s" totalImagesPullingTime="443.736444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:54.093742684 +0000 UTC m=+622.443470085" watchObservedRunningTime="2026-10-30 21:11:54.095028719 +0000 UTC m=+622.444756117"
Oct 30 21:11:54.840779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:54.840744    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:11:54.841093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:54.841071    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:11:54.991417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:54.991382    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:11:55.044073 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:11:55.044021    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:11:55.837698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:55.833801    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:11:55.837698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:55.833872    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:11:57.046695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:57.045847    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:11:58.573970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:11:58.573920    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:12:00.092299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:00.092272    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:12:00.159354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:00.159299    4633 generic.go:356] "Generic (PLEG): container finished" podID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerID="b992455bdf9e7e4ab1d167e7d8d63c57cf084d05bbab877b475ead547ac2f1e3" exitCode=0
Oct 30 21:12:00.159538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:00.159446    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" event={"ID":"5778ea70-23d5-467d-bd3d-e034ef9327dc","Type":"ContainerDied","Data":"b992455bdf9e7e4ab1d167e7d8d63c57cf084d05bbab877b475ead547ac2f1e3"}
Oct 30 21:12:00.160271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:00.159952    4633 scope.go:122] "RemoveContainer" containerID="b992455bdf9e7e4ab1d167e7d8d63c57cf084d05bbab877b475ead547ac2f1e3"
Oct 30 21:12:00.161448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:00.161417    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:00.161530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:00.161488    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:00.161568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:00.161527    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:01.172962 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:01.172916    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" event={"ID":"5778ea70-23d5-467d-bd3d-e034ef9327dc","Type":"ContainerStarted","Data":"4857af0cab47837ec01a777def7f51fe7a7197eb4d30bb85a7a0edb65c97e27b"}
Oct 30 21:12:01.338711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:01.338137    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:12:01.338711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:01.338276    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:12:01.338711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:01.338319    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:12:01.890841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:01.890809    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:12:01.891027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:01.890858    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:12:02.341739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:02.324137    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:12:02.341739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:02.324201    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:12:03.066833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.066798    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:12:03.068533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.068505    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:12:03.068648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.068596    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:12:03.190605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.190571    4633 generic.go:356] "Generic (PLEG): container finished" podID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerID="47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865" exitCode=0
Oct 30 21:12:03.190812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.190744    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" event={"ID":"d87731e5-07ec-4842-8e0a-f98200c5acf3","Type":"ContainerDied","Data":"47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865"}
Oct 30 21:12:03.191104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.191090    4633 scope.go:122] "RemoveContainer" containerID="47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865"
Oct 30 21:12:03.192574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.192546    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:03.192646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.192615    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:03.192711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.192648    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:03.317516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.317028    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:12:03.317516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:03.317097    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:12:04.200878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:04.200837    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" event={"ID":"d87731e5-07ec-4842-8e0a-f98200c5acf3","Type":"ContainerStarted","Data":"058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271"}
Oct 30 21:12:04.324696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:04.321103    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:12:04.324696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:04.321168    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:12:04.871089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:04.871052    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:12:04.871441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:04.871418    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:12:04.871588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:04.871565    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:12:05.317078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:05.317040    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:12:05.321701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:05.317632    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:12:05.871165 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:05.871125    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:12:05.871323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:05.871179    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:12:06.330696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:06.319069    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:12:06.332067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:06.331598    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:12:06.400703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:06.400652    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:12:06.870434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:06.870396    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:12:06.870661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:06.870452    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:12:07.447243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:07.447194    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:12:07.448956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:07.448935    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:12:07.450767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:07.450715    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:12:07.871289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:07.871208    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:12:07.871289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:07.871268    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:12:08.290863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.290824    4633 generic.go:356] "Generic (PLEG): container finished" podID="ae638011-2505-459d-ad37-e3e618d710dc" containerID="18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43" exitCode=0
Oct 30 21:12:08.291076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.290897    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" event={"ID":"ae638011-2505-459d-ad37-e3e618d710dc","Type":"ContainerDied","Data":"18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43"}
Oct 30 21:12:08.291435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.291415    4633 scope.go:122] "RemoveContainer" containerID="18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43"
Oct 30 21:12:08.292512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.292487    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:08.292577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.292553    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:08.292615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.292583    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:08.871067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.871029    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:12:08.871508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.871088    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:12:08.969698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.961053    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"]
Oct 30 21:12:08.980843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.978430    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"]
Oct 30 21:12:08.980843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.978549    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:08.984923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.984893    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:12:08.985148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.985014    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:12:08.985148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.985087    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:12:08.985148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:08.985143    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z426l\""
Oct 30 21:12:09.174429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.174378    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.174429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.174430    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.174758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.174477    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.174758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.174532    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.174758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.174553    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5fk\" (UniqueName: \"kubernetes.io/projected/2a4dcdda-32c7-4337-9d26-7de4e0edd073-kube-api-access-7q5fk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.174758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.174577    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4dcdda-32c7-4337-9d26-7de4e0edd073-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.174758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.174615    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.275860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.275824    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.275860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.275865    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5fk\" (UniqueName: \"kubernetes.io/projected/2a4dcdda-32c7-4337-9d26-7de4e0edd073-kube-api-access-7q5fk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.276138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.275891    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4dcdda-32c7-4337-9d26-7de4e0edd073-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.276138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.275929    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.276138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.275951    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.276138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.275981    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.276138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.276025    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.277185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.277159    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.278589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.278566    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.278929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.278906    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.279205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.279188    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.281891 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.281848    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4dcdda-32c7-4337-9d26-7de4e0edd073-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.283592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.283571    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5fk\" (UniqueName: \"kubernetes.io/projected/2a4dcdda-32c7-4337-9d26-7de4e0edd073-kube-api-access-7q5fk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.284371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.284347    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4dcdda-32c7-4337-9d26-7de4e0edd073-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6pl4g\" (UID: \"2a4dcdda-32c7-4337-9d26-7de4e0edd073\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.296554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.296533    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:09.299928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.299899    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" event={"ID":"ae638011-2505-459d-ad37-e3e618d710dc","Type":"ContainerStarted","Data":"b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898"}
Oct 30 21:12:09.666995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.666885    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"]
Oct 30 21:12:09.669300 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:12:09.669267    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4dcdda_32c7_4337_9d26_7de4e0edd073.slice/crio-4c421d50072920707d5ddbd290485881fedeae0ac3fce5b25881589ea36ce815 WatchSource:0}: Error finding container 4c421d50072920707d5ddbd290485881fedeae0ac3fce5b25881589ea36ce815: Status 404 returned error can't find the container with id 4c421d50072920707d5ddbd290485881fedeae0ac3fce5b25881589ea36ce815
Oct 30 21:12:09.808947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.808852    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:12:09.809241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.809218    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:09.809314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.809276    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:12:09.877349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.874930    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:12:09.877349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.876193    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:12:09.877826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:09.877379    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:12:10.308515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:10.308483    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g" event={"ID":"2a4dcdda-32c7-4337-9d26-7de4e0edd073","Type":"ContainerStarted","Data":"4c421d50072920707d5ddbd290485881fedeae0ac3fce5b25881589ea36ce815"}
Oct 30 21:12:10.809327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:10.809297    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:10.809500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:10.809359    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:12:11.809302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:11.809264    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:11.809805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:11.809356    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:12:12.494499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:12.494449    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:12:12.809972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:12.809343    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:12.809972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:12.809407    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:12:13.809627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:13.809584    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:12:13.809831 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:13.809649    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:12:14.181736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:14.181700    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:14.182140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:14.181778    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:12:14.375233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:14.375186    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g" event={"ID":"2a4dcdda-32c7-4337-9d26-7de4e0edd073","Type":"ContainerStarted","Data":"19988a1a0822898ac05fd14f360ddc909c92ae46dbdca4ce06633fafcc43a653"}
Oct 30 21:12:14.375412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:14.375279    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:14.400043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:14.399993    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g" podStartSLOduration=1.892671323 podStartE2EDuration="6.399979811s" totalImagesPullingTime="4.507308488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:14.396253134 +0000 UTC m=+642.745980530" watchObservedRunningTime="2026-10-30 21:12:14.399979811 +0000 UTC m=+642.749707206"
Oct 30 21:12:14.814060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:14.814027    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:12:14.814601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:14.814572    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:12:14.815441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:14.815419    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:12:15.383818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:15.383791    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6pl4g"
Oct 30 21:12:21.265148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:21.265076    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:12:25.017832 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:12:25.017789    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:12:28.606367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:28.606327    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:12:39.402974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:39.402941    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:12:55.014394 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:12:55.014303    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:12:57.978196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:12:57.978161    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:13:09.407392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:09.407352    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:13:19.374903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.374857    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"]
Oct 30 21:13:19.379184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.379160    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.390103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.390080    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a33b12-openshift-default-dockercfg-6bm8g\""
Oct 30 21:13:19.392884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.392860    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"]
Oct 30 21:13:19.456755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.456717    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ddf97db-429a-47db-b503-56559682eb9c-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.456917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.456766    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.456917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.456851    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.456917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.456903    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.457043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.457016    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k7mw\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-kube-api-access-6k7mw\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.457080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.457066    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.457129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.457121    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.457166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.457137    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ddf97db-429a-47db-b503-56559682eb9c-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.457213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.457165    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.558626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.558579    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.558843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.558685    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k7mw\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-kube-api-access-6k7mw\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.558843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.558725    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.558843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.558754    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.558843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.558780    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ddf97db-429a-47db-b503-56559682eb9c-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.558923    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.559082    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.559115    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ddf97db-429a-47db-b503-56559682eb9c-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.559143    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.559168    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.559232    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.559259    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.559535    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ddf97db-429a-47db-b503-56559682eb9c-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.559714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.559682    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.561318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.561296    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.561717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.561661    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ddf97db-429a-47db-b503-56559682eb9c-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.567166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.567131    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.568319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.568294    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k7mw\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-kube-api-access-6k7mw\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:19.700345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:19.699979    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:20.049180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:20.049078    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"]
Oct 30 21:13:20.154940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:20.154885    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:20.155241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:20.155213    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:20.155412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:20.155392    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:20.874129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:20.874085    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" event={"ID":"9ddf97db-429a-47db-b503-56559682eb9c","Type":"ContainerStarted","Data":"bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0"}
Oct 30 21:13:20.874635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:20.874152    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" event={"ID":"9ddf97db-429a-47db-b503-56559682eb9c","Type":"ContainerStarted","Data":"4ca4ec6558509e8f2aecf3b9b650cb2f4e0f09cef149bfce99067f7db81bc2a3"}
Oct 30 21:13:20.899978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:20.899929    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" podStartSLOduration=1.899914879 podStartE2EDuration="1.899914879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:20.895843659 +0000 UTC m=+709.245571054" watchObservedRunningTime="2026-10-30 21:13:20.899914879 +0000 UTC m=+709.249642275"
Oct 30 21:13:21.700273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:21.700233    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:21.701990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:21.701968    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:13:21.702053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:21.702017    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:13:22.701185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:22.701145    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:13:22.701703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:22.701209    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:13:23.700525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:23.700486    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:13:23.700736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:23.700541    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:13:24.700577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:24.700543    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:13:24.701050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:24.700611    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:13:25.044145 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:13:25.043992    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:13:25.700431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:25.700399    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:13:25.700604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:25.700464    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:13:26.701197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:26.701162    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:13:26.701718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:26.701220    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:13:27.706136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:27.706106    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:27.706519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:27.706467    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:27.707188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:27.707169    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:13:29.457570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.457535    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"]
Oct 30 21:13:29.461808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.461786    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.471367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.471344    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:13:29.471530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.471390    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:13:29.471642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.471458    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-22787fe12174\""
Oct 30 21:13:29.471642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.471593    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:13:29.472176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.471653    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bf44c1ed9483\""
Oct 30 21:13:29.472176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.471808    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-v9w8x\""
Oct 30 21:13:29.472176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.472069    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:13:29.474152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.474130    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"]
Oct 30 21:13:29.605512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.604366    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-plugins-config-volume\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.605512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.604443    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-provider-credentials\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.605512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.604486    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthm4\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-kube-api-access-hthm4\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.605512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.604553    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-routing-overlay-volume\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.705819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.705770    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-plugins-config-volume\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.706067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.705839    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-provider-credentials\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.706067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.705874    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthm4\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-kube-api-access-hthm4\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.706067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.705944    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-routing-overlay-volume\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.706617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.706589    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-plugins-config-volume\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.706782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.706758    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-routing-overlay-volume\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.710113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.710042    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-provider-credentials\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.715832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.715784    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthm4\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-kube-api-access-hthm4\") pod \"payload-processing-external-model-75dccdd755-dqlgl\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") " pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:29.778998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:29.778960    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:30.073270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:30.073175    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"]
Oct 30 21:13:30.093048 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:13:30.091540    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a148523_fabc_4df3_babd_56e44d5b5354.slice/crio-38287f939e3f0a58cd1e949ddd8f65f24b98f4b8bfa4462f5379d2d50509ad48 WatchSource:0}: Error finding container 38287f939e3f0a58cd1e949ddd8f65f24b98f4b8bfa4462f5379d2d50509ad48: Status 404 returned error can't find the container with id 38287f939e3f0a58cd1e949ddd8f65f24b98f4b8bfa4462f5379d2d50509ad48
Oct 30 21:13:30.988652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:30.988604    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl" event={"ID":"2a148523-fabc-4df3-babd-56e44d5b5354","Type":"ContainerStarted","Data":"87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac"}
Oct 30 21:13:30.989225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:30.988697    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl" event={"ID":"2a148523-fabc-4df3-babd-56e44d5b5354","Type":"ContainerStarted","Data":"38287f939e3f0a58cd1e949ddd8f65f24b98f4b8bfa4462f5379d2d50509ad48"}
Oct 30 21:13:30.989225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:30.988952    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:31.006647 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:31.006591    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl" podStartSLOduration=2.006576105 podStartE2EDuration="2.006576105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:31.004819833 +0000 UTC m=+719.354547240" watchObservedRunningTime="2026-10-30 21:13:31.006576105 +0000 UTC m=+719.356303507"
Oct 30 21:13:35.965853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:35.965816    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:13:38.959847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:38.959801    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:13:38.960610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:38.960569    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:13:38.960873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:38.960851    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:13:38.961024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:38.961002    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:13:41.999817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:41.999780    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:13:45.147462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.147421    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"]
Oct 30 21:13:45.151661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.151636    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.160586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.160561    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-384164-openshift-default-dockercfg-qs2n4\""
Oct 30 21:13:45.164565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.164536    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"]
Oct 30 21:13:45.266805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.266766    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.266805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.266812    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.267115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.266836    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.267115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.266976    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45f95ab-4f37-4976-8512-0fc08613ecef-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.267115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.267012    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.267115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.267043    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.267263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.267161    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp98m\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-kube-api-access-dp98m\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.267263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.267181    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.267263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.267198    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368283    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45f95ab-4f37-4976-8512-0fc08613ecef-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368331    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368453    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368568    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp98m\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-kube-api-access-dp98m\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368590    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368605    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368643    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368719    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368731    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.368826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.368762    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.369060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.369001    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.369130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.369086    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.369130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.369093    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.369738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.369716    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45f95ab-4f37-4976-8512-0fc08613ecef-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.371168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.371146    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.371268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.371241    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.377528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.377506    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.377617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.377593    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp98m\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-kube-api-access-dp98m\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-lglfc\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.466653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.466620    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:45.756413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.756381    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"]
Oct 30 21:13:45.756550 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:13:45.756525    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45f95ab_4f37_4976_8512_0fc08613ecef.slice/crio-1dbf51a0eaac5f0693823680fe2c38cc01765c2c9b9f4636ce7604fada67ec51 WatchSource:0}: Error finding container 1dbf51a0eaac5f0693823680fe2c38cc01765c2c9b9f4636ce7604fada67ec51: Status 404 returned error can't find the container with id 1dbf51a0eaac5f0693823680fe2c38cc01765c2c9b9f4636ce7604fada67ec51
Oct 30 21:13:45.759008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.758976    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:45.759094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.759038    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:45.759094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:45.759067    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:13:46.636349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:46.636306    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" event={"ID":"c45f95ab-4f37-4976-8512-0fc08613ecef","Type":"ContainerStarted","Data":"e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32"}
Oct 30 21:13:46.637041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:46.637015    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" event={"ID":"c45f95ab-4f37-4976-8512-0fc08613ecef","Type":"ContainerStarted","Data":"1dbf51a0eaac5f0693823680fe2c38cc01765c2c9b9f4636ce7604fada67ec51"}
Oct 30 21:13:46.683731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:46.683635    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podStartSLOduration=1.683612263 podStartE2EDuration="1.683612263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:46.677580715 +0000 UTC m=+735.027308117" watchObservedRunningTime="2026-10-30 21:13:46.683612263 +0000 UTC m=+735.033339659"
Oct 30 21:13:47.478122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:47.478084    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:47.478556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:47.478531    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:47.496213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:47.496148    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:13:48.470697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:48.467067    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:48.470697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:48.467142    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:13:49.467242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:49.467204    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:49.467587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:49.467520    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:13:49.814963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:49.814835    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:13:49.815731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:49.815695    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:13:50.469130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:50.469089    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:50.469875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:50.469440    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:13:51.467834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:51.467781    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:51.468240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:51.467853    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:13:52.164449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:52.164416    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:13:52.467623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:52.467539    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:13:52.467623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:52.467595    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 21:13:53.564196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:53.564159    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:53.564604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:53.564582    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:53.565411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:13:53.565386    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:13:55.015076 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:13:55.015040    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:14:04.367210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:04.367178    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"]
Oct 30 21:14:05.623161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:05.623124    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:14:08.749617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:08.749582    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-5xcps"]
Oct 30 21:14:08.756395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:08.756368    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:08.871191 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:08.871156    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-5xcps"]
Oct 30 21:14:08.939141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:08.939102    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-routing-overlay-volume\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:08.939316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:08.939205    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t82z\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-kube-api-access-2t82z\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:08.939316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:08.939237    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-provider-credentials\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:08.939405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:08.939364    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-plugins-config-volume\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.040054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.039964    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-routing-overlay-volume\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.040054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.040027    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t82z\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-kube-api-access-2t82z\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.040258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.040139    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-provider-credentials\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.040258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.040247    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-plugins-config-volume\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.040655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.040629    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-routing-overlay-volume\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.040934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.040852    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-plugins-config-volume\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.042884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.042853    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-provider-credentials\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.048850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.048825    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t82z\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-kube-api-access-2t82z\") pod \"payload-processing-external-model-698f9c649-5xcps\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.146557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.146523    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:09.368270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.368159    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl" podUID="2a148523-fabc-4df3-babd-56e44d5b5354" containerName="payload-processing" containerID="cri-o://87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac" gracePeriod=25
Oct 30 21:14:09.427085 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.427057    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-5xcps"]
Oct 30 21:14:09.450735 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:14:09.450700    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7416fe_d5c9_4eaf_9b48_8e3fd030595a.slice/crio-108ffc8447496df874aea5fb6546adbeba8db3d528a208cb561d045060dfbf0b WatchSource:0}: Error finding container 108ffc8447496df874aea5fb6546adbeba8db3d528a208cb561d045060dfbf0b: Status 404 returned error can't find the container with id 108ffc8447496df874aea5fb6546adbeba8db3d528a208cb561d045060dfbf0b
Oct 30 21:14:09.619600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.619540    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:14:09.646742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.646714    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-routing-overlay-volume\") pod \"2a148523-fabc-4df3-babd-56e44d5b5354\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") "
Oct 30 21:14:09.646911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.646781    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-kube-api-access-hthm4\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-kube-api-access-hthm4\") pod \"2a148523-fabc-4df3-babd-56e44d5b5354\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") "
Oct 30 21:14:09.646911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.646828    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-plugins-config-volume\") pod \"2a148523-fabc-4df3-babd-56e44d5b5354\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") "
Oct 30 21:14:09.647021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.646971    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-provider-credentials\") pod \"2a148523-fabc-4df3-babd-56e44d5b5354\" (UID: \"2a148523-fabc-4df3-babd-56e44d5b5354\") "
Oct 30 21:14:09.647223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.647197    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-routing-overlay-volume" pod "2a148523-fabc-4df3-babd-56e44d5b5354" (UID: "2a148523-fabc-4df3-babd-56e44d5b5354"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:09.647324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.647294    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-plugins-config-volume" pod "2a148523-fabc-4df3-babd-56e44d5b5354" (UID: "2a148523-fabc-4df3-babd-56e44d5b5354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:09.647413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.647393    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:09.647455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.647420    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a148523-fabc-4df3-babd-56e44d5b5354-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:09.649639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.649603    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-provider-credentials" pod "2a148523-fabc-4df3-babd-56e44d5b5354" (UID: "2a148523-fabc-4df3-babd-56e44d5b5354"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:09.649778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.649759    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-kube-api-access-hthm4" pod "2a148523-fabc-4df3-babd-56e44d5b5354" (UID: "2a148523-fabc-4df3-babd-56e44d5b5354"). InnerVolumeSpecName "kube-api-access-hthm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:09.748466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.748429    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hthm4\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-kube-api-access-hthm4\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:09.748466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:09.748459    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a148523-fabc-4df3-babd-56e44d5b5354-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:10.205254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.205218    4633 generic.go:356] "Generic (PLEG): container finished" podID="2a148523-fabc-4df3-babd-56e44d5b5354" containerID="87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac" exitCode=0
Oct 30 21:14:10.205712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.205261    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"
Oct 30 21:14:10.205712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.205301    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl" event={"ID":"2a148523-fabc-4df3-babd-56e44d5b5354","Type":"ContainerDied","Data":"87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac"}
Oct 30 21:14:10.205712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.205355    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl" event={"ID":"2a148523-fabc-4df3-babd-56e44d5b5354","Type":"ContainerDied","Data":"38287f939e3f0a58cd1e949ddd8f65f24b98f4b8bfa4462f5379d2d50509ad48"}
Oct 30 21:14:10.205712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.205377    4633 scope.go:122] "RemoveContainer" containerID="87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac"
Oct 30 21:14:10.208163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.208135    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps" event={"ID":"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a","Type":"ContainerStarted","Data":"b10a989f408054c988525b9f4b1469b87d36e398d44ceae0a65006640a232b6f"}
Oct 30 21:14:10.208327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.208185    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps" event={"ID":"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a","Type":"ContainerStarted","Data":"108ffc8447496df874aea5fb6546adbeba8db3d528a208cb561d045060dfbf0b"}
Oct 30 21:14:10.208327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.208296    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:10.217083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.217059    4633 scope.go:122] "RemoveContainer" containerID="87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac"
Oct 30 21:14:10.217340 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:14:10.217320    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac\": container with ID starting with 87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac not found: ID does not exist" containerID="87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac"
Oct 30 21:14:10.217446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.217347    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac"} err="failed to get container status \"87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac\": rpc error: code = NotFound desc = could not find container \"87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac\": container with ID starting with 87d5737bcf750f39d786293a6cff02c17df253b004bf264631379a4659edbaac not found: ID does not exist"
Oct 30 21:14:10.227230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.227179    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps" podStartSLOduration=2.227167196 podStartE2EDuration="2.227167196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:10.224727292 +0000 UTC m=+758.574454690" watchObservedRunningTime="2026-10-30 21:14:10.227167196 +0000 UTC m=+758.576894591"
Oct 30 21:14:10.240287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.240260    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"]
Oct 30 21:14:10.245614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:10.245590    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75dccdd755-dqlgl"]
Oct 30 21:14:11.796956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:11.796919    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a148523-fabc-4df3-babd-56e44d5b5354" path="/var/lib/kubelet/pods/2a148523-fabc-4df3-babd-56e44d5b5354/volumes"
Oct 30 21:14:12.754698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.754592    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"]
Oct 30 21:14:12.755162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.755149    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2a148523-fabc-4df3-babd-56e44d5b5354" containerName="payload-processing"
Oct 30 21:14:12.755205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.755164    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a148523-fabc-4df3-babd-56e44d5b5354" containerName="payload-processing"
Oct 30 21:14:12.755244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.755238    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a148523-fabc-4df3-babd-56e44d5b5354" containerName="payload-processing"
Oct 30 21:14:12.760105 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.760081    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776060    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776114    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776115    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"]
Oct 30 21:14:12.776249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776143    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47db1ff-15b0-4942-b38f-84e0c342659f-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776286    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776315    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776441    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776478    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776499    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4nt\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-kube-api-access-mp4nt\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.776715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.776601    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.878865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.878794    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.878865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.878863    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.889979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.878896    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4nt\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-kube-api-access-mp4nt\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.889979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.878927    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.889979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.878954    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.889979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.879015    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.889979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.879056    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47db1ff-15b0-4942-b38f-84e0c342659f-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.889979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.879184    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.889979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.879214    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.889979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.889416    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.890404 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.890111    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47db1ff-15b0-4942-b38f-84e0c342659f-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.892495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.891178    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.892495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.892178    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.894048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.892599    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.896019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.895995    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.896953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.896926    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.901009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.900984    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:12.901121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:12.901098    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4nt\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-kube-api-access-mp4nt\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:13.076341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:13.076237    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:13.362405 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:14:13.362380    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47db1ff_15b0_4942_b38f_84e0c342659f.slice/crio-1f242e554633f2d3227bdccbd1029438269d4e76574e3dae7614bbf16c4be724 WatchSource:0}: Error finding container 1f242e554633f2d3227bdccbd1029438269d4e76574e3dae7614bbf16c4be724: Status 404 returned error can't find the container with id 1f242e554633f2d3227bdccbd1029438269d4e76574e3dae7614bbf16c4be724
Oct 30 21:14:13.362547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:13.362523    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"]
Oct 30 21:14:13.365236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:13.365205    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:14:13.365316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:13.365269    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:14:13.365316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:13.365300    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:14:14.246147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:14.246111    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" event={"ID":"f47db1ff-15b0-4942-b38f-84e0c342659f","Type":"ContainerStarted","Data":"d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78"}
Oct 30 21:14:14.246568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:14.246170    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" event={"ID":"f47db1ff-15b0-4942-b38f-84e0c342659f","Type":"ContainerStarted","Data":"1f242e554633f2d3227bdccbd1029438269d4e76574e3dae7614bbf16c4be724"}
Oct 30 21:14:14.270091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:14.270035    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podStartSLOduration=2.27001796 podStartE2EDuration="2.27001796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:14.267228315 +0000 UTC m=+762.616955753" watchObservedRunningTime="2026-10-30 21:14:14.27001796 +0000 UTC m=+762.619745356"
Oct 30 21:14:15.076387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:15.076348    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:15.078307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:15.078279    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:14:15.078455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:15.078327    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 21:14:16.076861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:16.076826    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:14:16.077364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:16.076879    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 21:14:17.080808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:17.079108    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:14:17.080808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:17.079178    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 21:14:18.088816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:18.088777    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:14:18.089419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:18.089388    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 21:14:19.080696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:19.080064    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:14:19.080696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:19.080128    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 21:14:20.088793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:20.088757    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:14:20.089427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:20.089390    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 21:14:21.234998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:21.234959    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:22.120836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:22.120791    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:14:22.121199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:22.121164    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:14:22.667732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:22.667650    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:22.668268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:22.668232    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:22.669126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:22.669104    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:14:23.486777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:23.486734    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:14:24.336835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.336794    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-558f755994-hxjzs"]
Oct 30 21:14:24.345877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.345182    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.361880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.350838    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f755994-hxjzs"]
Oct 30 21:14:24.361880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.354115    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a6465e7287a2\""
Oct 30 21:14:24.361880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.354768    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5e066e5cb288\""
Oct 30 21:14:24.446646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.446600    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs28p\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-kube-api-access-gs28p\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.446914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.446762    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-plugins-config-volume\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.446914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.446832    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-provider-credentials\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.446914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.446873    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-routing-overlay-volume\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.548141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.548104    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs28p\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-kube-api-access-gs28p\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.548361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.548209    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-plugins-config-volume\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.548361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.548254    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-provider-credentials\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.548361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.548281    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-routing-overlay-volume\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.550292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.549631    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-plugins-config-volume\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.550292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.549739    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-routing-overlay-volume\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.563232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.563197    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-provider-credentials\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.572764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.572733    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs28p\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-kube-api-access-gs28p\") pod \"payload-processing-external-model-558f755994-hxjzs\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") " pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:24.721914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:24.721877    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:25.019011 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:14:25.018912    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:14:25.041740 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:14:25.041702    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf551d236_1b63_449f_b463_56434666631e.slice/crio-a8d5884c23ae554069c42aad0935ffc435d7f38fc883d059c16f2f86e9143edb WatchSource:0}: Error finding container a8d5884c23ae554069c42aad0935ffc435d7f38fc883d059c16f2f86e9143edb: Status 404 returned error can't find the container with id a8d5884c23ae554069c42aad0935ffc435d7f38fc883d059c16f2f86e9143edb
Oct 30 21:14:25.043040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:25.042776    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f755994-hxjzs"]
Oct 30 21:14:25.512459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:25.512420    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs" event={"ID":"f551d236-1b63-449f-b463-56434666631e","Type":"ContainerStarted","Data":"620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0"}
Oct 30 21:14:25.512999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:25.512481    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs" event={"ID":"f551d236-1b63-449f-b463-56434666631e","Type":"ContainerStarted","Data":"a8d5884c23ae554069c42aad0935ffc435d7f38fc883d059c16f2f86e9143edb"}
Oct 30 21:14:25.512999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:25.512638    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:25.530270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:25.530206    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs" podStartSLOduration=1.5301917999999999 podStartE2EDuration="1.5301918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:25.528899616 +0000 UTC m=+773.878627012" watchObservedRunningTime="2026-10-30 21:14:25.5301918 +0000 UTC m=+773.879919197"
Oct 30 21:14:36.522374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:36.522341    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:14:36.566977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:36.566941    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-5xcps"]
Oct 30 21:14:40.204329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:40.204291    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:14:41.568696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:41.567828    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps" podUID="bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" containerName="payload-processing" containerID="cri-o://b10a989f408054c988525b9f4b1469b87d36e398d44ceae0a65006640a232b6f" gracePeriod=25
Oct 30 21:14:41.768296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:41.767973    4633 generic.go:356] "Generic (PLEG): container finished" podID="bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" containerID="b10a989f408054c988525b9f4b1469b87d36e398d44ceae0a65006640a232b6f" exitCode=0
Oct 30 21:14:41.768296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:41.768198    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps" event={"ID":"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a","Type":"ContainerDied","Data":"b10a989f408054c988525b9f4b1469b87d36e398d44ceae0a65006640a232b6f"}
Oct 30 21:14:42.562017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.561966    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:42.691701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.691130    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-kube-api-access-2t82z\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-kube-api-access-2t82z\") pod \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") "
Oct 30 21:14:42.691701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.691210    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-routing-overlay-volume\") pod \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") "
Oct 30 21:14:42.691701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.691251    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-provider-credentials\") pod \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") "
Oct 30 21:14:42.691701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.691382    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-plugins-config-volume\") pod \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\" (UID: \"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a\") "
Oct 30 21:14:42.706796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.705697    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-routing-overlay-volume" pod "bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" (UID: "bf7416fe-d5c9-4eaf-9b48-8e3fd030595a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:42.706796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.705914    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-plugins-config-volume" pod "bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" (UID: "bf7416fe-d5c9-4eaf-9b48-8e3fd030595a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:42.773954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.772254    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-provider-credentials" pod "bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" (UID: "bf7416fe-d5c9-4eaf-9b48-8e3fd030595a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:42.777915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.777881    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-kube-api-access-2t82z" pod "bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" (UID: "bf7416fe-d5c9-4eaf-9b48-8e3fd030595a"). InnerVolumeSpecName "kube-api-access-2t82z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:42.800000 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.799965    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:42.800232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.800212    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:42.800338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.800324    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t82z\" (UniqueName: \"kubernetes.io/projected/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-kube-api-access-2t82z\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:42.800446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.800432    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:42.802152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.802120    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps" event={"ID":"bf7416fe-d5c9-4eaf-9b48-8e3fd030595a","Type":"ContainerDied","Data":"108ffc8447496df874aea5fb6546adbeba8db3d528a208cb561d045060dfbf0b"}
Oct 30 21:14:42.802332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.802317    4633 scope.go:122] "RemoveContainer" containerID="b10a989f408054c988525b9f4b1469b87d36e398d44ceae0a65006640a232b6f"
Oct 30 21:14:42.802627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.802609    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698f9c649-5xcps"
Oct 30 21:14:42.937751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.934952    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-5xcps"]
Oct 30 21:14:42.948704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:42.947013    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-5xcps"]
Oct 30 21:14:43.895697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:43.893586    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" path="/var/lib/kubelet/pods/bf7416fe-d5c9-4eaf-9b48-8e3fd030595a/volumes"
Oct 30 21:14:55.019915 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:14:55.019876    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:14:56.204480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.204443    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"]
Oct 30 21:14:56.205764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.205738    4633 cpu_manager.go:414] "containerMap: removing container" podUID="bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" containerName="payload-processing"
Oct 30 21:14:56.205764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.205764    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" containerName="payload-processing"
Oct 30 21:14:56.206075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.205898    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7416fe-d5c9-4eaf-9b48-8e3fd030595a" containerName="payload-processing"
Oct 30 21:14:56.209755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.209733    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.214807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.214511    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"openshift-service-ca.crt\""
Oct 30 21:14:56.214807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.214698    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"kube-root-ca.crt\""
Oct 30 21:14:56.214807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.214778    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"default-dockercfg-45nrp\""
Oct 30 21:14:56.215054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.214896    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"distinct-2-w2-05a342-kserve-self-signed-certs\""
Oct 30 21:14:56.230432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.230401    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"]
Oct 30 21:14:56.243317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.243284    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-model-cache\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.243595 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.243574    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-dshm\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.243711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.243633    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-home\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.243861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.243833    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqkxg\" (UniqueName: \"kubernetes.io/projected/20ce0d97-730b-4936-959b-795acbd830be-kube-api-access-pqkxg\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.243942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.243925    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ce0d97-730b-4936-959b-795acbd830be-tls-certs\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.243981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.243957    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-tmp-dir\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.351072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.347395    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-model-cache\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.351072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.347538    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-dshm\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.351072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.347592    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-home\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.351072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.347651    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqkxg\" (UniqueName: \"kubernetes.io/projected/20ce0d97-730b-4936-959b-795acbd830be-kube-api-access-pqkxg\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.351072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.347751    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ce0d97-730b-4936-959b-795acbd830be-tls-certs\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.351072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.347775    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-tmp-dir\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.351072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.348270    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-tmp-dir\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.351072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.348543    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-model-cache\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.379693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.360196    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-home\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.394696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.386282    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ce0d97-730b-4936-959b-795acbd830be-tls-certs\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.394696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.387605    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-dshm\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.394696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.390759    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqkxg\" (UniqueName: \"kubernetes.io/projected/20ce0d97-730b-4936-959b-795acbd830be-kube-api-access-pqkxg\") pod \"distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.531737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.531641    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:56.873486 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:14:56.867549    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20ce0d97_730b_4936_959b_795acbd830be.slice/crio-513e4bbdff03a37b221444868aa640d081221a634280a54fff44d5e31bbaf538 WatchSource:0}: Error finding container 513e4bbdff03a37b221444868aa640d081221a634280a54fff44d5e31bbaf538: Status 404 returned error can't find the container with id 513e4bbdff03a37b221444868aa640d081221a634280a54fff44d5e31bbaf538
Oct 30 21:14:56.878174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:56.878130    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"]
Oct 30 21:14:57.007757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:57.007726    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" event={"ID":"20ce0d97-730b-4936-959b-795acbd830be","Type":"ContainerStarted","Data":"44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6"}
Oct 30 21:14:57.007929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:57.007766    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" event={"ID":"20ce0d97-730b-4936-959b-795acbd830be","Type":"ContainerStarted","Data":"513e4bbdff03a37b221444868aa640d081221a634280a54fff44d5e31bbaf538"}
Oct 30 21:14:57.532134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:57.532096    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:14:59.776188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.776138    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podStartSLOduration=3.7761234740000003 podStartE2EDuration="3.776123474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:57.030439616 +0000 UTC m=+805.380167012" watchObservedRunningTime="2026-10-30 21:14:59.776123474 +0000 UTC m=+808.125850869"
Oct 30 21:14:59.776830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.776813    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"]
Oct 30 21:14:59.780815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.780798    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.794357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.794330    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"]
Oct 30 21:14:59.805882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.805859    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.806022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.805892    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.806101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.806075    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dld\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-kube-api-access-q9dld\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.806160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.806133    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.806223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.806187    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.806280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.806267    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.806342 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.806304    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.806411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.806392    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.806941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.806924    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.907971    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908015    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908136    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908225    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908275    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908308    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908404    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dld\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-kube-api-access-q9dld\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908419    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908446    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908485    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908496    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908622    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908807    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.908994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.908975    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.910914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.910889    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.911069 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.911055    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.916225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.916197    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:14:59.916383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:14:59.916368    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dld\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-kube-api-access-q9dld\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:15:00.098856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.098781    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:15:00.160102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.160070    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"]
Oct 30 21:15:00.165485 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.165444    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:00.185365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.185333    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:15:00.192145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.192116    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"]
Oct 30 21:15:00.200935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.200898    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"]
Oct 30 21:15:00.207554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.207516    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"]
Oct 30 21:15:00.207751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.207701    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:00.212799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.212765    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92266572-d6cb-4d15-9e8f-64d6ed9fa657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n\" (UID: \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:00.212944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.212857    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6p7\" (UniqueName: \"kubernetes.io/projected/92266572-d6cb-4d15-9e8f-64d6ed9fa657-kube-api-access-vf6p7\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n\" (UID: \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:00.314214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.314184    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287\" (UID: \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:00.314391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.314364    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwvt9\" (UniqueName: \"kubernetes.io/projected/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-kube-api-access-vwvt9\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287\" (UID: \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:00.314570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.314546    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92266572-d6cb-4d15-9e8f-64d6ed9fa657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n\" (UID: \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:00.314653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.314637    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6p7\" (UniqueName: \"kubernetes.io/projected/92266572-d6cb-4d15-9e8f-64d6ed9fa657-kube-api-access-vf6p7\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n\" (UID: \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:00.315203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.315182    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92266572-d6cb-4d15-9e8f-64d6ed9fa657-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n\" (UID: \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:00.324467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.324434    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6p7\" (UniqueName: \"kubernetes.io/projected/92266572-d6cb-4d15-9e8f-64d6ed9fa657-kube-api-access-vf6p7\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n\" (UID: \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:00.389421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.389396    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"]
Oct 30 21:15:00.389844 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:15:00.389819    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef74e2dc_1fb5_414d_b969_5edc2fb704ef.slice/crio-fe5dfc7e6d2d373d4b37abde98016ead6bb247019efb2d0c7cd53f27abbaab2d WatchSource:0}: Error finding container fe5dfc7e6d2d373d4b37abde98016ead6bb247019efb2d0c7cd53f27abbaab2d: Status 404 returned error can't find the container with id fe5dfc7e6d2d373d4b37abde98016ead6bb247019efb2d0c7cd53f27abbaab2d
Oct 30 21:15:00.392351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.392320    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:00.392417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.392383    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:00.392417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.392412    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:00.415654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.415630    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287\" (UID: \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:00.415816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.415722    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwvt9\" (UniqueName: \"kubernetes.io/projected/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-kube-api-access-vwvt9\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287\" (UID: \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:00.416253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.416231    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287\" (UID: \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:00.423299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.423277    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwvt9\" (UniqueName: \"kubernetes.io/projected/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-kube-api-access-vwvt9\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287\" (UID: \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:00.493943 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.493907    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:00.526709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.526653    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:00.791213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.791185    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"]
Oct 30 21:15:00.791737 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:15:00.791706    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92266572_d6cb_4d15_9e8f_64d6ed9fa657.slice/crio-cec11cebe5179c5c529349a60cb9644db5343f34a0efaada9b37736021ad72b9 WatchSource:0}: Error finding container cec11cebe5179c5c529349a60cb9644db5343f34a0efaada9b37736021ad72b9: Status 404 returned error can't find the container with id cec11cebe5179c5c529349a60cb9644db5343f34a0efaada9b37736021ad72b9
Oct 30 21:15:00.835195 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:00.835161    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"]
Oct 30 21:15:00.836946 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:15:00.836922    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e0d70e_339c_4224_9037_7a7a3f8aedf5.slice/crio-10599871e7680ba4691f72722bab3bacdba42591ea18bc8ee7a5ea14497eb05a WatchSource:0}: Error finding container 10599871e7680ba4691f72722bab3bacdba42591ea18bc8ee7a5ea14497eb05a: Status 404 returned error can't find the container with id 10599871e7680ba4691f72722bab3bacdba42591ea18bc8ee7a5ea14497eb05a
Oct 30 21:15:01.076624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:01.076294    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" event={"ID":"92266572-d6cb-4d15-9e8f-64d6ed9fa657","Type":"ContainerStarted","Data":"cec11cebe5179c5c529349a60cb9644db5343f34a0efaada9b37736021ad72b9"}
Oct 30 21:15:01.077850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:01.077824    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" event={"ID":"b4e0d70e-339c-4224-9037-7a7a3f8aedf5","Type":"ContainerStarted","Data":"10599871e7680ba4691f72722bab3bacdba42591ea18bc8ee7a5ea14497eb05a"}
Oct 30 21:15:01.079856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:01.079816    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" event={"ID":"ef74e2dc-1fb5-414d-b969-5edc2fb704ef","Type":"ContainerStarted","Data":"cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda"}
Oct 30 21:15:01.079981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:01.079881    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" event={"ID":"ef74e2dc-1fb5-414d-b969-5edc2fb704ef","Type":"ContainerStarted","Data":"fe5dfc7e6d2d373d4b37abde98016ead6bb247019efb2d0c7cd53f27abbaab2d"}
Oct 30 21:15:01.102510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:01.102008    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:15:01.106138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:01.105809    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:15:01.106138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:01.105877    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 21:15:01.121689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:01.118148    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podStartSLOduration=2.118126689 podStartE2EDuration="2.118126689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.109220557 +0000 UTC m=+809.458947959" watchObservedRunningTime="2026-10-30 21:15:01.118126689 +0000 UTC m=+809.467854086"
Oct 30 21:15:02.099646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:02.099614    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:15:02.100086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:02.099691    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 21:15:03.100975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:03.099947    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:15:03.100975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:03.100006    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 21:15:04.099467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:04.099429    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:15:04.099695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:04.099490    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 21:15:05.099173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:05.099135    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:15:05.099619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:05.099193    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 21:15:06.099353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:06.099262    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:15:06.099353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:06.099317    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 21:15:06.147797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:06.147763    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" event={"ID":"b4e0d70e-339c-4224-9037-7a7a3f8aedf5","Type":"ContainerStarted","Data":"26a361bd11196e68325b0e0743e54b83ddb053b947f3948c6fc0121139a7697d"}
Oct 30 21:15:06.149544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:06.149518    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" event={"ID":"92266572-d6cb-4d15-9e8f-64d6ed9fa657","Type":"ContainerStarted","Data":"88ec0284d9a58315dcafb7acff3f080535d254b20414a5f2d0bf6605222e88e3"}
Oct 30 21:15:06.168041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:06.167986    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" podStartSLOduration=1.567528198 podStartE2EDuration="6.167973323s" totalImagesPullingTime="4.600445125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:06.164991278 +0000 UTC m=+814.514718675" watchObservedRunningTime="2026-10-30 21:15:06.167973323 +0000 UTC m=+814.517700719"
Oct 30 21:15:06.185588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:06.185535    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" podStartSLOduration=1.789674099 podStartE2EDuration="6.185519816s" totalImagesPullingTime="4.395845717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:06.184625262 +0000 UTC m=+814.534352660" watchObservedRunningTime="2026-10-30 21:15:06.185519816 +0000 UTC m=+814.535247214"
Oct 30 21:15:06.541913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:06.540193    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:15:06.554535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:06.554505    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:15:07.105383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.105339    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:15:07.106712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.106473    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:15:07.106712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.106536    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 21:15:07.166977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.166939    4633 generic.go:356] "Generic (PLEG): container finished" podID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerID="26a361bd11196e68325b0e0743e54b83ddb053b947f3948c6fc0121139a7697d" exitCode=60
Oct 30 21:15:07.167544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.167513    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" event={"ID":"b4e0d70e-339c-4224-9037-7a7a3f8aedf5","Type":"ContainerDied","Data":"26a361bd11196e68325b0e0743e54b83ddb053b947f3948c6fc0121139a7697d"}
Oct 30 21:15:07.168263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.168245    4633 scope.go:122] "RemoveContainer" containerID="26a361bd11196e68325b0e0743e54b83ddb053b947f3948c6fc0121139a7697d"
Oct 30 21:15:07.174135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.174106    4633 generic.go:356] "Generic (PLEG): container finished" podID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerID="88ec0284d9a58315dcafb7acff3f080535d254b20414a5f2d0bf6605222e88e3" exitCode=60
Oct 30 21:15:07.174247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.174191    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" event={"ID":"92266572-d6cb-4d15-9e8f-64d6ed9fa657","Type":"ContainerDied","Data":"88ec0284d9a58315dcafb7acff3f080535d254b20414a5f2d0bf6605222e88e3"}
Oct 30 21:15:07.174506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.174487    4633 scope.go:122] "RemoveContainer" containerID="88ec0284d9a58315dcafb7acff3f080535d254b20414a5f2d0bf6605222e88e3"
Oct 30 21:15:07.193027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:07.192438    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:15:08.188466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.188392    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" event={"ID":"92266572-d6cb-4d15-9e8f-64d6ed9fa657","Type":"ContainerStarted","Data":"fed706c9e4736f53dd5eed649c717a5f049c713280624d409854ecc63cb7fe25"}
Oct 30 21:15:08.191438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.191401    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" event={"ID":"b4e0d70e-339c-4224-9037-7a7a3f8aedf5","Type":"ContainerStarted","Data":"cc1cab3a57a398a741829bf1b99a9662dde6731a1e1a7faa9ceece53a6a2ad36"}
Oct 30 21:15:08.355412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.345975    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"]
Oct 30 21:15:08.377752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.373529    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.378767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.378740    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"]
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412359    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412414    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmp5\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-kube-api-access-djmp5\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412458    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412498    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412554    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412706    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412763    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412787    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.413690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.412832    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.514702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.514416    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.514702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.514492    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.514702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.514560    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.514702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.514590    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmp5\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-kube-api-access-djmp5\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.514702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.514642    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.525880    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.526631    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.526741    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.526929    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.526996    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.527332    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.528033    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.528275    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.529698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.528500    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.554737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.553899    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.554737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.554138    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.564553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.564520    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmp5\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-kube-api-access-djmp5\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.604698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.591210    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:08.766141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:08.765991    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:09.166696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.161792    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:09.166696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.161872    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:09.486066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.448301    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:15:09.486066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.449525    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:15:09.486066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.468947    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:15:09.590399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.590361    4633 generic.go:356] "Generic (PLEG): container finished" podID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerID="fed706c9e4736f53dd5eed649c717a5f049c713280624d409854ecc63cb7fe25" exitCode=60
Oct 30 21:15:09.590779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.590751    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" event={"ID":"92266572-d6cb-4d15-9e8f-64d6ed9fa657","Type":"ContainerDied","Data":"fed706c9e4736f53dd5eed649c717a5f049c713280624d409854ecc63cb7fe25"}
Oct 30 21:15:09.590977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.590960    4633 scope.go:122] "RemoveContainer" containerID="88ec0284d9a58315dcafb7acff3f080535d254b20414a5f2d0bf6605222e88e3"
Oct 30 21:15:09.592717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.592692    4633 scope.go:122] "RemoveContainer" containerID="fed706c9e4736f53dd5eed649c717a5f049c713280624d409854ecc63cb7fe25"
Oct 30 21:15:09.593086 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:15:09.593054    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n_odh-ai-gateway-infra(92266572-d6cb-4d15-9e8f-64d6ed9fa657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657"
Oct 30 21:15:09.609392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.608818    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"]
Oct 30 21:15:09.622353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.614764    4633 generic.go:356] "Generic (PLEG): container finished" podID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerID="cc1cab3a57a398a741829bf1b99a9662dde6731a1e1a7faa9ceece53a6a2ad36" exitCode=60
Oct 30 21:15:09.622353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.614867    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" event={"ID":"b4e0d70e-339c-4224-9037-7a7a3f8aedf5","Type":"ContainerDied","Data":"cc1cab3a57a398a741829bf1b99a9662dde6731a1e1a7faa9ceece53a6a2ad36"}
Oct 30 21:15:09.622353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.615218    4633 scope.go:122] "RemoveContainer" containerID="cc1cab3a57a398a741829bf1b99a9662dde6731a1e1a7faa9ceece53a6a2ad36"
Oct 30 21:15:09.622353 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:15:09.615457    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287_odh-ai-gateway-infra(b4e0d70e-339c-4224-9037-7a7a3f8aedf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5"
Oct 30 21:15:09.702579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.702552    4633 scope.go:122] "RemoveContainer" containerID="26a361bd11196e68325b0e0743e54b83ddb053b947f3948c6fc0121139a7697d"
Oct 30 21:15:09.703552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.703502    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:09.703719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.703601    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:09.703719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:09.703651    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:10.623928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:10.623899    4633 scope.go:122] "RemoveContainer" containerID="fed706c9e4736f53dd5eed649c717a5f049c713280624d409854ecc63cb7fe25"
Oct 30 21:15:10.624413 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:15:10.624236    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n_odh-ai-gateway-infra(92266572-d6cb-4d15-9e8f-64d6ed9fa657)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657"
Oct 30 21:15:10.625590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:10.625544    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" event={"ID":"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a","Type":"ContainerStarted","Data":"d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f"}
Oct 30 21:15:10.625734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:10.625611    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" event={"ID":"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a","Type":"ContainerStarted","Data":"dcb84d326de756d8abe2641d3e3ba27384df2bf9dce6baca55b8d498cc4c6101"}
Oct 30 21:15:10.628438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:10.628412    4633 scope.go:122] "RemoveContainer" containerID="cc1cab3a57a398a741829bf1b99a9662dde6731a1e1a7faa9ceece53a6a2ad36"
Oct 30 21:15:10.628775 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:15:10.628755    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287_odh-ai-gateway-infra(b4e0d70e-339c-4224-9037-7a7a3f8aedf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5"
Oct 30 21:15:10.661979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:10.661912    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podStartSLOduration=2.66189176 podStartE2EDuration="2.66189176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:10.657997752 +0000 UTC m=+819.007725149" watchObservedRunningTime="2026-10-30 21:15:10.66189176 +0000 UTC m=+819.011619204"
Oct 30 21:15:10.766219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:10.766184    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:10.767858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:10.767833    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:15:10.767983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:10.767885    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 21:15:11.767411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:11.767377    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:15:11.767892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:11.767433    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 21:15:12.767189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:12.767147    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:15:12.767417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:12.767240    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 21:15:13.767354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:13.767321    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:15:13.767800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:13.767378    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 21:15:14.766496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:14.766456    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:15:14.766694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:14.766529    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 21:15:15.084640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:15.084523    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f755994-hxjzs"]
Oct 30 21:15:15.766937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:15.766899    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:15:15.767122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:15.766969    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 21:15:16.766953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:16.766921    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:15:16.767416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:16.766979    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 21:15:18.105704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:18.105649    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:18.106196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:18.106064    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:18.106826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:18.106806    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:15:19.783406 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:19.783369    4633 scope.go:122] "RemoveContainer" containerID="cc1cab3a57a398a741829bf1b99a9662dde6731a1e1a7faa9ceece53a6a2ad36"
Oct 30 21:15:19.783900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:19.783535    4633 scope.go:122] "RemoveContainer" containerID="fed706c9e4736f53dd5eed649c717a5f049c713280624d409854ecc63cb7fe25"
Oct 30 21:15:20.085300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.085205    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs" podUID="f551d236-1b63-449f-b463-56434666631e" containerName="payload-processing" containerID="cri-o://620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0" gracePeriod=25
Oct 30 21:15:20.484382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.484343    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:15:20.652719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.640310    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-routing-overlay-volume\") pod \"f551d236-1b63-449f-b463-56434666631e\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") "
Oct 30 21:15:20.660768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.653174    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-kube-api-access-gs28p\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-kube-api-access-gs28p\") pod \"f551d236-1b63-449f-b463-56434666631e\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") "
Oct 30 21:15:20.664751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.656885    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-routing-overlay-volume" pod "f551d236-1b63-449f-b463-56434666631e" (UID: "f551d236-1b63-449f-b463-56434666631e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.685814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.665319    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-plugins-config-volume\") pod \"f551d236-1b63-449f-b463-56434666631e\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") "
Oct 30 21:15:20.686226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.686185    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-provider-credentials\") pod \"f551d236-1b63-449f-b463-56434666631e\" (UID: \"f551d236-1b63-449f-b463-56434666631e\") "
Oct 30 21:15:20.687095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.687073    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.699564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.665279    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-plugins-config-volume" pod "f551d236-1b63-449f-b463-56434666631e" (UID: "f551d236-1b63-449f-b463-56434666631e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.700027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.699951    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-kube-api-access-gs28p" pod "f551d236-1b63-449f-b463-56434666631e" (UID: "f551d236-1b63-449f-b463-56434666631e"). InnerVolumeSpecName "kube-api-access-gs28p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.728091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.724260    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-provider-credentials" pod "f551d236-1b63-449f-b463-56434666631e" (UID: "f551d236-1b63-449f-b463-56434666631e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.791419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.791388    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs28p\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-kube-api-access-gs28p\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.791958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.791940    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f551d236-1b63-449f-b463-56434666631e-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.792050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.792038    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f551d236-1b63-449f-b463-56434666631e-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:20.794703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.794051    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" event={"ID":"92266572-d6cb-4d15-9e8f-64d6ed9fa657","Type":"ContainerStarted","Data":"1eade1e822459ff22bc938d5185e8fb6c687e676b79bfcc6f7104f91a07aaff5"}
Oct 30 21:15:20.798451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.797529    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" event={"ID":"b4e0d70e-339c-4224-9037-7a7a3f8aedf5","Type":"ContainerStarted","Data":"1289a7705cf75e2e5f3ee03837577e58f9c8bf3da1c0db155258fc1854c26e8b"}
Oct 30 21:15:20.798451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.797878    4633 scope.go:122] "RemoveContainer" containerID="1289a7705cf75e2e5f3ee03837577e58f9c8bf3da1c0db155258fc1854c26e8b"
Oct 30 21:15:20.798451 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:15:20.798111    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287_odh-ai-gateway-infra(b4e0d70e-339c-4224-9037-7a7a3f8aedf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5"
Oct 30 21:15:20.821725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.818825    4633 generic.go:356] "Generic (PLEG): container finished" podID="f551d236-1b63-449f-b463-56434666631e" containerID="620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0" exitCode=0
Oct 30 21:15:20.821725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.818897    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs" event={"ID":"f551d236-1b63-449f-b463-56434666631e","Type":"ContainerDied","Data":"620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0"}
Oct 30 21:15:20.821725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.818949    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs" event={"ID":"f551d236-1b63-449f-b463-56434666631e","Type":"ContainerDied","Data":"a8d5884c23ae554069c42aad0935ffc435d7f38fc883d059c16f2f86e9143edb"}
Oct 30 21:15:20.821725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.818978    4633 scope.go:122] "RemoveContainer" containerID="620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0"
Oct 30 21:15:20.821725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.819180    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-558f755994-hxjzs"
Oct 30 21:15:20.973245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.973221    4633 scope.go:122] "RemoveContainer" containerID="620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0"
Oct 30 21:15:20.974130 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:15:20.974106    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0\": container with ID starting with 620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0 not found: ID does not exist" containerID="620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0"
Oct 30 21:15:20.974295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:20.974270    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0"} err="failed to get container status \"620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0\": rpc error: code = NotFound desc = could not find container \"620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0\": container with ID starting with 620aa32867a99fba90ef46db726acbf67b2593a0a4e8aceed86f5c7fffa260b0 not found: ID does not exist"
Oct 30 21:15:21.006697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:21.006646    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f755994-hxjzs"]
Oct 30 21:15:21.009924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:21.009888    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-558f755994-hxjzs"]
Oct 30 21:15:21.851773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:21.851304    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f551d236-1b63-449f-b463-56434666631e" path="/var/lib/kubelet/pods/f551d236-1b63-449f-b463-56434666631e/volumes"
Oct 30 21:15:21.914077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:21.914021    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"]
Oct 30 21:15:21.966697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:21.964326    4633 generic.go:356] "Generic (PLEG): container finished" podID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerID="1eade1e822459ff22bc938d5185e8fb6c687e676b79bfcc6f7104f91a07aaff5" exitCode=60
Oct 30 21:15:21.966697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:21.964776    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" event={"ID":"92266572-d6cb-4d15-9e8f-64d6ed9fa657","Type":"ContainerDied","Data":"1eade1e822459ff22bc938d5185e8fb6c687e676b79bfcc6f7104f91a07aaff5"}
Oct 30 21:15:21.966697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:21.964944    4633 scope.go:122] "RemoveContainer" containerID="fed706c9e4736f53dd5eed649c717a5f049c713280624d409854ecc63cb7fe25"
Oct 30 21:15:22.011564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:22.011528    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"]
Oct 30 21:15:22.185705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:22.185181    4633 generic.go:356] "Generic (PLEG): container finished" podID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerID="1289a7705cf75e2e5f3ee03837577e58f9c8bf3da1c0db155258fc1854c26e8b" exitCode=60
Oct 30 21:15:22.185705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:22.185336    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" event={"ID":"b4e0d70e-339c-4224-9037-7a7a3f8aedf5","Type":"ContainerDied","Data":"1289a7705cf75e2e5f3ee03837577e58f9c8bf3da1c0db155258fc1854c26e8b"}
Oct 30 21:15:22.825074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:22.825040    4633 scope.go:122] "RemoveContainer" containerID="cc1cab3a57a398a741829bf1b99a9662dde6731a1e1a7faa9ceece53a6a2ad36"
Oct 30 21:15:23.803068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.802713    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:23.909728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.906912    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-md8nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:23.909728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.906968    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:23.948708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.932868    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92266572-d6cb-4d15-9e8f-64d6ed9fa657-kube-api-access-vf6p7\" (UniqueName: \"kubernetes.io/projected/92266572-d6cb-4d15-9e8f-64d6ed9fa657-kube-api-access-vf6p7\") pod \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\" (UID: \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\") "
Oct 30 21:15:23.948708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.933060    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92266572-d6cb-4d15-9e8f-64d6ed9fa657-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92266572-d6cb-4d15-9e8f-64d6ed9fa657-maas-api-service-ca\") pod \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\" (UID: \"92266572-d6cb-4d15-9e8f-64d6ed9fa657\") "
Oct 30 21:15:23.968985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.956912    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92266572-d6cb-4d15-9e8f-64d6ed9fa657-maas-api-service-ca" pod "92266572-d6cb-4d15-9e8f-64d6ed9fa657" (UID: "92266572-d6cb-4d15-9e8f-64d6ed9fa657"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:23.968985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.958287    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:23.968985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.958340    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:23.968985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.968809    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:23.968985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:23.968899    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:24.040939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.040905    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92266572-d6cb-4d15-9e8f-64d6ed9fa657-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:24.067887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.067327    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92266572-d6cb-4d15-9e8f-64d6ed9fa657-kube-api-access-vf6p7" pod "92266572-d6cb-4d15-9e8f-64d6ed9fa657" (UID: "92266572-d6cb-4d15-9e8f-64d6ed9fa657"). InnerVolumeSpecName "kube-api-access-vf6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:24.099105 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.099065    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n" event={"ID":"92266572-d6cb-4d15-9e8f-64d6ed9fa657","Type":"ContainerDied","Data":"cec11cebe5179c5c529349a60cb9644db5343f34a0efaada9b37736021ad72b9"}
Oct 30 21:15:24.099379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.099360    4633 scope.go:122] "RemoveContainer" containerID="1eade1e822459ff22bc938d5185e8fb6c687e676b79bfcc6f7104f91a07aaff5"
Oct 30 21:15:24.103383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.103358    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:24.110716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.103898    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"
Oct 30 21:15:24.213288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.213250    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6p7\" (UniqueName: \"kubernetes.io/projected/92266572-d6cb-4d15-9e8f-64d6ed9fa657-kube-api-access-vf6p7\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:24.669948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.669919    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"]
Oct 30 21:15:24.687696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.681768    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889915-z4f7n"]
Oct 30 21:15:24.710692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.707351    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-kube-api-access-vwvt9\" (UniqueName: \"kubernetes.io/projected/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-kube-api-access-vwvt9\") pod \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\" (UID: \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\") "
Oct 30 21:15:24.710692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.707455    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-maas-api-service-ca\") pod \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\" (UID: \"b4e0d70e-339c-4224-9037-7a7a3f8aedf5\") "
Oct 30 21:15:24.710692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.709156    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-maas-api-service-ca" pod "b4e0d70e-339c-4224-9037-7a7a3f8aedf5" (UID: "b4e0d70e-339c-4224-9037-7a7a3f8aedf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:24.729412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.729377    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-kube-api-access-vwvt9" pod "b4e0d70e-339c-4224-9037-7a7a3f8aedf5" (UID: "b4e0d70e-339c-4224-9037-7a7a3f8aedf5"). InnerVolumeSpecName "kube-api-access-vwvt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:24.810111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.810011    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwvt9\" (UniqueName: \"kubernetes.io/projected/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-kube-api-access-vwvt9\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:24.810111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:24.810050    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4e0d70e-339c-4224-9037-7a7a3f8aedf5-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:25.016881 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:15:25.016797    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:15:25.112603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:25.112579    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"
Oct 30 21:15:25.112603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:25.112583    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287" event={"ID":"b4e0d70e-339c-4224-9037-7a7a3f8aedf5","Type":"ContainerDied","Data":"10599871e7680ba4691f72722bab3bacdba42591ea18bc8ee7a5ea14497eb05a"}
Oct 30 21:15:25.112874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:25.112682    4633 scope.go:122] "RemoveContainer" containerID="1289a7705cf75e2e5f3ee03837577e58f9c8bf3da1c0db155258fc1854c26e8b"
Oct 30 21:15:25.146035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:25.146004    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"]
Oct 30 21:15:25.152718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:25.152648    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889915-qp287"]
Oct 30 21:15:25.791357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:25.791326    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" path="/var/lib/kubelet/pods/92266572-d6cb-4d15-9e8f-64d6ed9fa657/volumes"
Oct 30 21:15:25.791782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:25.791768    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" path="/var/lib/kubelet/pods/b4e0d70e-339c-4224-9037-7a7a3f8aedf5/volumes"
Oct 30 21:15:27.862089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862061    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"]
Oct 30 21:15:27.862600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862587    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f551d236-1b63-449f-b463-56434666631e" containerName="payload-processing"
Oct 30 21:15:27.862640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862601    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f551d236-1b63-449f-b463-56434666631e" containerName="payload-processing"
Oct 30 21:15:27.862640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862614    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.862640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862619    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.862640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862634    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.862640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862639    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862648    4633 cpu_manager.go:414] "containerMap: removing container" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862653    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862660    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862678    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862684    4633 cpu_manager.go:414] "containerMap: removing container" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862688    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862788    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f551d236-1b63-449f-b463-56434666631e" containerName="payload-processing"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862796    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862803    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862809    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862816    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.862825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.862823    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.866468 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.866451    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.883112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.883074    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"]
Oct 30 21:15:27.905201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.905177    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"]
Oct 30 21:15:27.906061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.906034    4633 cpu_manager.go:414] "containerMap: removing container" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.906061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.906060    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92266572-d6cb-4d15-9e8f-64d6ed9fa657" containerName="cleanup"
Oct 30 21:15:27.906246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.906225    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e0d70e-339c-4224-9037-7a7a3f8aedf5" containerName="cleanup"
Oct 30 21:15:27.913145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.913122    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:27.933492 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.933466    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"]
Oct 30 21:15:27.960359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960327    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.960493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960381    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31437c22-0870-430e-8ab4-195e4b44d622-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.960493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960474    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.960599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960519    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.960662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960597    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31437c22-0870-430e-8ab4-195e4b44d622-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.960756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960711    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.960756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960748    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.960873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960859    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9sr\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-kube-api-access-td9sr\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:27.960913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:27.960897    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.061768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.061735    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.061768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.061768    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.061790    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.061813    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvfmz\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-kube-api-access-cvfmz\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.061832    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.061900    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.061975    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.061996    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9sr\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-kube-api-access-td9sr\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062026    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062131    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062162    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062208    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062239    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062262    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062352    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.062600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062461    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062490    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31437c22-0870-430e-8ab4-195e4b44d622-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062496    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062531    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062573    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062612    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31437c22-0870-430e-8ab4-195e4b44d622-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.062902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.062888    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.063263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.063245    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31437c22-0870-430e-8ab4-195e4b44d622-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.065025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.064992    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.065249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.065229    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31437c22-0870-430e-8ab4-195e4b44d622-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.069824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.069799    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.069916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.069862    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9sr\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-kube-api-access-td9sr\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.163931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.163900    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.163948    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.163977    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.163997    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164026    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164181    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164216    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvfmz\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-kube-api-access-cvfmz\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164242    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164271    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164574    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164614    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164765    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.164821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.164791    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.165170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.165152    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.166849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.166822    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.166947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.166857    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.173085 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.173059    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.179466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.179439    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvfmz\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-kube-api-access-cvfmz\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.182208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.182187    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:28.235580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.235544    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:28.492838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.492806    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"]
Oct 30 21:15:28.493304 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:15:28.493284    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31437c22_0870_430e_8ab4_195e4b44d622.slice/crio-625ecaf7a7579514b289f9e1e5785342d4e0469efa68da5373317d2c72c753fa WatchSource:0}: Error finding container 625ecaf7a7579514b289f9e1e5785342d4e0469efa68da5373317d2c72c753fa: Status 404 returned error can't find the container with id 625ecaf7a7579514b289f9e1e5785342d4e0469efa68da5373317d2c72c753fa
Oct 30 21:15:28.495683 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.495634    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:28.495771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.495732    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:28.495771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.495763    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:28.560836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.560809    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"]
Oct 30 21:15:28.561817 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:15:28.561793    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b1e6f9_5eae_4485_849b_2209b534f9eb.slice/crio-533ed3f54f10ba865585a0ee9098d5fd75a0f81fe200f699b0101752b678b4da WatchSource:0}: Error finding container 533ed3f54f10ba865585a0ee9098d5fd75a0f81fe200f699b0101752b678b4da: Status 404 returned error can't find the container with id 533ed3f54f10ba865585a0ee9098d5fd75a0f81fe200f699b0101752b678b4da
Oct 30 21:15:28.564389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.564349    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:28.564461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.564419    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:28.564461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:28.564450    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:29.167662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.167612    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" event={"ID":"31437c22-0870-430e-8ab4-195e4b44d622","Type":"ContainerStarted","Data":"438a8562538b4b0ddda04d58f6263b61c914ad700647c0d65e0c897da7ef15e4"}
Oct 30 21:15:29.168239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.167731    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" event={"ID":"31437c22-0870-430e-8ab4-195e4b44d622","Type":"ContainerStarted","Data":"625ecaf7a7579514b289f9e1e5785342d4e0469efa68da5373317d2c72c753fa"}
Oct 30 21:15:29.171695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.171467    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" event={"ID":"b0b1e6f9-5eae-4485-849b-2209b534f9eb","Type":"ContainerStarted","Data":"cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633"}
Oct 30 21:15:29.171695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.171526    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" event={"ID":"b0b1e6f9-5eae-4485-849b-2209b534f9eb","Type":"ContainerStarted","Data":"533ed3f54f10ba865585a0ee9098d5fd75a0f81fe200f699b0101752b678b4da"}
Oct 30 21:15:29.183371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.183343    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:29.185178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.185154    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:29.185264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.185205    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 21:15:29.191319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.191268    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podStartSLOduration=2.191252961 podStartE2EDuration="2.191252961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:29.190371507 +0000 UTC m=+837.540098904" watchObservedRunningTime="2026-10-30 21:15:29.191252961 +0000 UTC m=+837.540980361"
Oct 30 21:15:29.260694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.241821    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podStartSLOduration=2.241796065 podStartE2EDuration="2.241796065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:29.21153716 +0000 UTC m=+837.561264561" watchObservedRunningTime="2026-10-30 21:15:29.241796065 +0000 UTC m=+837.591523461"
Oct 30 21:15:29.265997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.248038    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:29.265997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.258825    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:15:29.265997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.262826    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:15:29.881693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.881143    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"]
Oct 30 21:15:29.895736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.895661    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"]
Oct 30 21:15:29.895933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.895822    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998240    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998291    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998416    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998488    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998598    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998650    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998743    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68sh\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-kube-api-access-g68sh\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998808    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:29.998931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:29.998890    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.100382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.100352    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.100538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.100395    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68sh\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-kube-api-access-g68sh\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.100646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.100623    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.100890 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.100870    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101039    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101161    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101201    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101262    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101336    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101414    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101445    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101741    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.101942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.101924    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.102096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.102081    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.103121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.103096    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.104358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.104337    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.108572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.108547    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68sh\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-kube-api-access-g68sh\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.109315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.109291    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.183786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.183757    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:30.184199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.183812    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 21:15:30.251217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.251153    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:15:30.251438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.251228    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:15:30.272620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.272584    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:30.671115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.671085    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"]
Oct 30 21:15:30.673906 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:15:30.673864    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0d2768_381d_4fa1_9d8f_d658fd1d3f69.slice/crio-86bbed78a41d43e9092b722dd2c56cc84c0d6f10e14f30ffca59b0d244680d4b WatchSource:0}: Error finding container 86bbed78a41d43e9092b722dd2c56cc84c0d6f10e14f30ffca59b0d244680d4b: Status 404 returned error can't find the container with id 86bbed78a41d43e9092b722dd2c56cc84c0d6f10e14f30ffca59b0d244680d4b
Oct 30 21:15:30.676929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.676887    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:30.677018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.676982    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:30.677062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:30.677034    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:31.183161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.183121    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:31.183469 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.183190    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 21:15:31.190718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.190654    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" event={"ID":"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69","Type":"ContainerStarted","Data":"a3ba535161755c60a2ba89b7a5f2660f87dd90e1a30fdd3774561180a0a1d038"}
Oct 30 21:15:31.191190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.190741    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" event={"ID":"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69","Type":"ContainerStarted","Data":"86bbed78a41d43e9092b722dd2c56cc84c0d6f10e14f30ffca59b0d244680d4b"}
Oct 30 21:15:31.211982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.211919    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podStartSLOduration=2.211902089 podStartE2EDuration="2.211902089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:31.2093063 +0000 UTC m=+839.559033723" watchObservedRunningTime="2026-10-30 21:15:31.211902089 +0000 UTC m=+839.561629485"
Oct 30 21:15:31.242695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.235963    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:15:31.242695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.236023    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:15:31.281697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.281590    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:31.281901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.281789    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:15:31.281901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:31.281855    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:15:32.184206 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:32.184166    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:32.184494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:32.184462    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 21:15:32.245697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:32.239104    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:15:32.245697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:32.239169    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:15:32.275785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:32.275471    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:15:32.275785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:32.275537    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:15:33.199813 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:33.199343    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:33.199813 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:33.199404    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 21:15:33.236476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:33.236219    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:15:33.236476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:33.236294    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:15:33.293693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:33.293272    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:15:33.294188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:33.293358    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:15:34.187698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.184088    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:34.187698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.184165    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 21:15:34.247697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.247092    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:15:34.247697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.247167    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:15:34.289713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.287874    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:15:34.289713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.287954    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:15:34.634781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.633779    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"]
Oct 30 21:15:34.654798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.652086    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.728704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.726456    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"]
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.779780    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.779843    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.779913    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.779944    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.780000    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvgv2\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-kube-api-access-wvgv2\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.780068    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.780110    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b81a30-63dd-45c4-b183-75a67e3f37b6-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.780142    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.785473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.780179    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.883702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.874324    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"]
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.889520    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.889603    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.889698    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.889730    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.889790    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvgv2\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-kube-api-access-wvgv2\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.889862    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.889908    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b81a30-63dd-45c4-b183-75a67e3f37b6-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.889944    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.890001    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.890605    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.906808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.892212    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.911927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.909097    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b81a30-63dd-45c4-b183-75a67e3f37b6-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.911927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.909382    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.911927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.910279    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.991318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.984797    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.991318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.987151    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvgv2\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-kube-api-access-wvgv2\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.991318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.989513    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:34.991318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.991068    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.991706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.991563    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:34.998984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:34.998182    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"]
Oct 30 21:15:35.093321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.093282    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.093708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.093684    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.093867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.093850    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.094050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.094023    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lsd4\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-kube-api-access-5lsd4\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.094140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.094074    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.094140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.094105    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.094140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.094131    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.094304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.094191    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.094304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.094263    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.184095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.183857    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:35.184095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.183945    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 21:15:35.195800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.195761    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.195800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.195793    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.195838    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lsd4\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-kube-api-access-5lsd4\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.195859    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.195877    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.195891    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.195932    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.195968    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.196023    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.196188    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.196414    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.196461    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.196708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.196686    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.197114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.197093    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.198791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.198768    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.199541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.199512    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.204585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.204557    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lsd4\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-kube-api-access-5lsd4\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.204801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.204781    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.236553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.236080    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:15:35.236553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.236144    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:15:35.273637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.273600    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:15:35.273868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.273695    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:15:35.289385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.289344    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:35.376156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.376117    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:35.666852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.665911    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:35.666852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.666004    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:35.666852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.666063    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:35.668746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.668240    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"]
Oct 30 21:15:35.796316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.796284    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"]
Oct 30 21:15:35.801295 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:15:35.801248    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0932eb_ba19_4b6f_9b2b_324572cd2aa0.slice/crio-e81af8b5aff4c7f9c1c6181c3b0ac49cfc1ab97df8660c95ceb2e8d6446d83fb WatchSource:0}: Error finding container e81af8b5aff4c7f9c1c6181c3b0ac49cfc1ab97df8660c95ceb2e8d6446d83fb: Status 404 returned error can't find the container with id e81af8b5aff4c7f9c1c6181c3b0ac49cfc1ab97df8660c95ceb2e8d6446d83fb
Oct 30 21:15:35.805039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.804841    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:35.805039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.804918    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:35.805039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:35.804948    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:36.183141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.183107    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:36.183443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.183399    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 21:15:36.243758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.240066    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:15:36.243758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.240125    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Oct 30 21:15:36.281696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.277252    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:15:36.281696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.277333    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:15:36.282640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.282608    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" event={"ID":"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0","Type":"ContainerStarted","Data":"0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815"}
Oct 30 21:15:36.282900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.282877    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" event={"ID":"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0","Type":"ContainerStarted","Data":"e81af8b5aff4c7f9c1c6181c3b0ac49cfc1ab97df8660c95ceb2e8d6446d83fb"}
Oct 30 21:15:36.312725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.312375    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" event={"ID":"16b81a30-63dd-45c4-b183-75a67e3f37b6","Type":"ContainerStarted","Data":"8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931"}
Oct 30 21:15:36.312725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.312495    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" event={"ID":"16b81a30-63dd-45c4-b183-75a67e3f37b6","Type":"ContainerStarted","Data":"9ffc36ac82726e766b77836afebbf4b4a5b7cbab99f0279e0cf500f2ab3b95bb"}
Oct 30 21:15:36.351693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.350737    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podStartSLOduration=2.350716863 podStartE2EDuration="2.350716863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:36.337079128 +0000 UTC m=+844.686806536" watchObservedRunningTime="2026-10-30 21:15:36.350716863 +0000 UTC m=+844.700444259"
Oct 30 21:15:36.383380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.383345    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:36.391744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:36.391445    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podStartSLOduration=2.391427928 podStartE2EDuration="2.391427928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:36.366266342 +0000 UTC m=+844.715993738" watchObservedRunningTime="2026-10-30 21:15:36.391427928 +0000 UTC m=+844.741155323"
Oct 30 21:15:37.289697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.286104    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:15:37.289697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.286168    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:15:37.291587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.291557    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:37.291915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.291890    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:37.292078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.292049    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:37.322694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.322607    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:37.329454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.324471    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:37.348834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.348759    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:15:37.417239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.412838    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:37.417239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.412905    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:37.778721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.775199    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:37.778721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.778161    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:37.793995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:37.793963    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:15:38.281162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.281126    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:15:38.281444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.281413    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:15:38.291760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.290751    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:38.291760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.290815    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:38.381035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.380927    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:38.381035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.380997    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:38.439697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.438947    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"]
Oct 30 21:15:38.481036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.477243    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.486110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.486071    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"]
Oct 30 21:15:38.603795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.603711    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.603795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.603774    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.603990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.603850    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.603990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.603872    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.603990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.603959    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.603990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.603978    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xskxh\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-kube-api-access-xskxh\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.604111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.604004    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.604111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.604043    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.604111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.604067    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.705698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.705375    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.705698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.705449    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xskxh\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-kube-api-access-xskxh\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.705698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.705600    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.705710    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.705752    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.705841    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.705893    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.706133    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.705895    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.706159    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.706191    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.706223    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.706709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.706684    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.707319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.707292    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.709148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.709123    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.710564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.710535    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.728360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.728288    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.734706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.729162    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xskxh\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-kube-api-access-xskxh\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:38.818214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:38.817914    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:39.299697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.295503    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:39.299697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.295564    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:39.405729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.405295    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:39.405729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.405348    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:39.507341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.507267    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:15:39.677375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.677343    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"]
Oct 30 21:15:39.677600 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:15:39.677326    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f6f617_37a2_45c6_b9ad_662c12df9fb5.slice/crio-245f5da2c19c979837eb4f7a27768392e8c989a65c8851ce9cd6cc55ff049d02 WatchSource:0}: Error finding container 245f5da2c19c979837eb4f7a27768392e8c989a65c8851ce9cd6cc55ff049d02: Status 404 returned error can't find the container with id 245f5da2c19c979837eb4f7a27768392e8c989a65c8851ce9cd6cc55ff049d02
Oct 30 21:15:39.682806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.682767    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:39.682903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.682853    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:39.682939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.682898    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:15:39.684605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.684587    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:39.686917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.686495    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:39.688711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:39.687605    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:15:40.290688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.290619    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:40.290882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.290715    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:40.377776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.377333    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:40.377776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.377428    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:40.474726 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.471096    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" event={"ID":"06f6f617-37a2-45c6-b9ad-662c12df9fb5","Type":"ContainerStarted","Data":"c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b"}
Oct 30 21:15:40.474726 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.471152    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" event={"ID":"06f6f617-37a2-45c6-b9ad-662c12df9fb5","Type":"ContainerStarted","Data":"245f5da2c19c979837eb4f7a27768392e8c989a65c8851ce9cd6cc55ff049d02"}
Oct 30 21:15:40.823375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.823343    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:40.823773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.823748    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:40.823970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:40.823943    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:41.297694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:41.290537    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:41.297694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:41.290621    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:41.384690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:41.378081    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:41.384690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:41.378146    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:41.828836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:41.826299    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:41.828836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:41.826371    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:42.299746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:42.295408    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:42.299746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:42.295480    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:42.402647 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:42.402615    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:42.403330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:42.403295    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:42.819695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:42.819035    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:42.819695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:42.819097    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:43.238695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:43.238149    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.337695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:43.337325    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:43.337695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:43.337385    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:43.441697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:43.441249    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:43.441697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:43.441315    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:43.858697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:43.830357    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:43.922804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:43.922754    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:44.282923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.254114    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.282923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.267649    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:15:44.319728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.310802    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:44.323836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.323782    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:44.482694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.465543    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podStartSLOduration=6.465519508 podStartE2EDuration="6.465519508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:40.542770033 +0000 UTC m=+848.892497430" watchObservedRunningTime="2026-10-30 21:15:44.465519508 +0000 UTC m=+852.815246904"
Oct 30 21:15:44.510953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.503061    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:44.510953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.503124    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:45.035770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.957551    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.035770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.957760    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:45.035770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:44.961242    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:15:45.424768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.415053    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.424768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.415118    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:45.424768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.415651    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.424768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.415735    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:45.950696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.942288    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.950696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.942351    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:45.977699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.968817    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:45.977699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.968890    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.977699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:45.968980    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.300256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:46.300157    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:46.300538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:46.300507    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:46.381596 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:46.381124    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:46.381596 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:46.381201    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:46.819091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:46.818623    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:46.819091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:46.818704    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:47.361211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:47.361025    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:47.361211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:47.361099    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:47.385834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:47.385717    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:47.385834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:47.385776    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:47.556831 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:47.556775    4633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.940715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:47.920445    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:47.940715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:47.920507    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:47.962372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:47.962330    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:48.290779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:48.290680    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:48.291031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:48.291000    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:48.393702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:48.390525    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:48.393702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:48.390583    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:48.830029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:48.829990    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:48.830424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:48.830361    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:49.289869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:49.289829    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:49.290062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:49.289896    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 30 21:15:49.398193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:49.394510    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:49.403815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:49.403747    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:15:49.821564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:49.821100    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:49.821564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:49.821166    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:50.289604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:50.289553    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:50.377548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:50.377513    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:50.574748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:50.571158    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:50.604036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:50.603949    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:15:50.833758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:50.819077    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:50.833758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:50.819141    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:51.151055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:51.151023    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:51.155482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:51.155457    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:15:51.820700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:51.819033    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:51.820700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:51.819091    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:52.818662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:52.818615    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:15:52.818902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:52.818706    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 21:15:53.818768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:53.818719    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:53.890570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:53.890519    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:54.543150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:15:54.543117    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:15:55.014620 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:15:55.014579    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:16:04.196412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:04.195845    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:04.196412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:04.196090    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:10.700916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:10.700881    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:16:15.043287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.043250    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"]
Oct 30 21:16:15.052408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.052379    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.084434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.084395    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"]
Oct 30 21:16:15.123233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123196    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.123434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123263    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bab83-0de4-4758-bd89-53f27be63c2d-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.123434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123406    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.123521 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123479    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.123565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123524    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.123613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123580    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6kmv\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-kube-api-access-w6kmv\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.123817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123795    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.123880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123826    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.123880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.123854    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225270    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6kmv\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-kube-api-access-w6kmv\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225392    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225413    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225437    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225562    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225647    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bab83-0de4-4758-bd89-53f27be63c2d-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225766    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225820    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.225862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225846    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.226028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225870    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.226028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225928    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.226028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.225995    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.226228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.226208    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.226454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.226432    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bab83-0de4-4758-bd89-53f27be63c2d-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.228165 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.228140    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.228382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.228357    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.233293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.233271    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.233513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.233489    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6kmv\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-kube-api-access-w6kmv\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.371524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.371437    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:15.727625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.727593    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"]
Oct 30 21:16:15.728417 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:16:15.728389    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6bab83_0de4_4758_bd89_53f27be63c2d.slice/crio-962a8b1cfa38f42c8143354167081de306e9628b87ff13ab10072f34337c492e WatchSource:0}: Error finding container 962a8b1cfa38f42c8143354167081de306e9628b87ff13ab10072f34337c492e: Status 404 returned error can't find the container with id 962a8b1cfa38f42c8143354167081de306e9628b87ff13ab10072f34337c492e
Oct 30 21:16:15.730916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.730878    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:15.731023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.730964    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:15.731023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:15.731010    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:16.496396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:16.496357    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" event={"ID":"ae6bab83-0de4-4758-bd89-53f27be63c2d","Type":"ContainerStarted","Data":"d7a721ffbd2bde2c87e29fdc3e73a9d909690100a3f6ef94ebc56e4de7dedba1"}
Oct 30 21:16:16.496939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:16.496429    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" event={"ID":"ae6bab83-0de4-4758-bd89-53f27be63c2d","Type":"ContainerStarted","Data":"962a8b1cfa38f42c8143354167081de306e9628b87ff13ab10072f34337c492e"}
Oct 30 21:16:17.371709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:17.371654    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:17.373396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:17.373369    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:16:17.373513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:17.373423    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 21:16:18.372700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:18.372631    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:16:18.373094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:18.372757    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 21:16:19.371823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:19.371779    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:16:19.372010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:19.371845    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 21:16:20.058201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.058132    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" podStartSLOduration=5.058114118 podStartE2EDuration="5.058114118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:16.540071333 +0000 UTC m=+884.889798730" watchObservedRunningTime="2026-10-30 21:16:20.058114118 +0000 UTC m=+888.407841515"
Oct 30 21:16:20.061794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.061770    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"]
Oct 30 21:16:20.074261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.074238    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.078628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.078597    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"]
Oct 30 21:16:20.078877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.078862    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:20.079970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.079948    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:16:20.158116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.148166    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.178911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.178871    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.180431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.180404    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hms8d\" (UniqueName: \"kubernetes.io/projected/63a57313-86cf-4b26-a3a6-4e3d660c674d-kube-api-access-hms8d\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.284295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.284254    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.284494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.284409    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hms8d\" (UniqueName: \"kubernetes.io/projected/63a57313-86cf-4b26-a3a6-4e3d660c674d-kube-api-access-hms8d\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.284494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.284459    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.284972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.284951    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.287045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.287019    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.293385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.293363    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hms8d\" (UniqueName: \"kubernetes.io/projected/63a57313-86cf-4b26-a3a6-4e3d660c674d-kube-api-access-hms8d\") pod \"maas-api-revoke-sub-1383e8ec-v259q\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.372588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.372508    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:16:20.372588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.372564    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 21:16:20.490588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.490550    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:20.843756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:20.842186    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"]
Oct 30 21:16:21.377698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:21.372734    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:16:21.377698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:21.372795    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 21:16:21.559121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:21.559038    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q" event={"ID":"63a57313-86cf-4b26-a3a6-4e3d660c674d","Type":"ContainerStarted","Data":"4c4c03f0bf672f146b15b660f5f68248910afdb9237fe1765edfca83ab1529ac"}
Oct 30 21:16:21.559121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:21.559097    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q" event={"ID":"63a57313-86cf-4b26-a3a6-4e3d660c674d","Type":"ContainerStarted","Data":"81ddc95a538689a852d2fd1c51f1c4523b2cf8d2b825c1aa9f5d6a9b03496b14"}
Oct 30 21:16:21.588269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:21.588188    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q" podStartSLOduration=1.588171027 podStartE2EDuration="1.588171027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:21.581591664 +0000 UTC m=+889.931319060" watchObservedRunningTime="2026-10-30 21:16:21.588171027 +0000 UTC m=+889.937898423"
Oct 30 21:16:22.372587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:22.372551    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:16:22.372785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:22.372606    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 21:16:22.568689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:22.568636    4633 generic.go:356] "Generic (PLEG): container finished" podID="63a57313-86cf-4b26-a3a6-4e3d660c674d" containerID="4c4c03f0bf672f146b15b660f5f68248910afdb9237fe1765edfca83ab1529ac" exitCode=0
Oct 30 21:16:22.569089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:22.568726    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q" event={"ID":"63a57313-86cf-4b26-a3a6-4e3d660c674d","Type":"ContainerDied","Data":"4c4c03f0bf672f146b15b660f5f68248910afdb9237fe1765edfca83ab1529ac"}
Oct 30 21:16:23.637633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.637603    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:23.638778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.638283    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:23.639345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.639325    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:16:23.780322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.780296    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:23.840007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.839912    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a57313-86cf-4b26-a3a6-4e3d660c674d-kube-api-access-hms8d\" (UniqueName: \"kubernetes.io/projected/63a57313-86cf-4b26-a3a6-4e3d660c674d-kube-api-access-hms8d\") pod \"63a57313-86cf-4b26-a3a6-4e3d660c674d\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") "
Oct 30 21:16:23.840189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.840013    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-xks-ca\") pod \"63a57313-86cf-4b26-a3a6-4e3d660c674d\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") "
Oct 30 21:16:23.840189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.840182    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-openshift-ca\") pod \"63a57313-86cf-4b26-a3a6-4e3d660c674d\" (UID: \"63a57313-86cf-4b26-a3a6-4e3d660c674d\") "
Oct 30 21:16:23.840611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.840579    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-openshift-ca" pod "63a57313-86cf-4b26-a3a6-4e3d660c674d" (UID: "63a57313-86cf-4b26-a3a6-4e3d660c674d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:23.840914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.840892    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-openshift-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:23.843819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.843778    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a57313-86cf-4b26-a3a6-4e3d660c674d-kube-api-access-hms8d" pod "63a57313-86cf-4b26-a3a6-4e3d660c674d" (UID: "63a57313-86cf-4b26-a3a6-4e3d660c674d"). InnerVolumeSpecName "kube-api-access-hms8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:23.843981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.843815    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-xks-ca" pod "63a57313-86cf-4b26-a3a6-4e3d660c674d" (UID: "63a57313-86cf-4b26-a3a6-4e3d660c674d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:23.942355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.942316    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hms8d\" (UniqueName: \"kubernetes.io/projected/63a57313-86cf-4b26-a3a6-4e3d660c674d-kube-api-access-hms8d\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:23.942355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:23.942356    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63a57313-86cf-4b26-a3a6-4e3d660c674d-maas-api-xks-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:24.602503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:24.602432    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"
Oct 30 21:16:24.606516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:24.602872    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q" event={"ID":"63a57313-86cf-4b26-a3a6-4e3d660c674d","Type":"ContainerDied","Data":"81ddc95a538689a852d2fd1c51f1c4523b2cf8d2b825c1aa9f5d6a9b03496b14"}
Oct 30 21:16:24.606808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:24.606781    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81ddc95a538689a852d2fd1c51f1c4523b2cf8d2b825c1aa9f5d6a9b03496b14"
Oct 30 21:16:25.017447 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:16:25.017406    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:16:27.546865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:27.546798    4633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:29.988850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:29.988813    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"]
Oct 30 21:16:30.005425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.005394    4633 cpu_manager.go:414] "containerMap: removing container" podUID="63a57313-86cf-4b26-a3a6-4e3d660c674d" containerName="revoke-keys"
Oct 30 21:16:30.005704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.005661    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a57313-86cf-4b26-a3a6-4e3d660c674d" containerName="revoke-keys"
Oct 30 21:16:30.006018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.006003    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a57313-86cf-4b26-a3a6-4e3d660c674d" containerName="revoke-keys"
Oct 30 21:16:30.223662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.223623    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"]
Oct 30 21:16:30.224105 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.224085    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.266753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.232641    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:16:30.323697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.278129    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:30.436696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.426162    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.456563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.442542    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.456563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.442693    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg72m\" (UniqueName: \"kubernetes.io/projected/830e6413-8bf5-46b8-a643-08f403366a81-kube-api-access-vg72m\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.546506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.546309    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.546875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.546849    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.547100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.547078    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg72m\" (UniqueName: \"kubernetes.io/projected/830e6413-8bf5-46b8-a643-08f403366a81-kube-api-access-vg72m\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.576156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.559555    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.604375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.584131    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.604613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.587465    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"]
Oct 30 21:16:30.612062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.609952    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg72m\" (UniqueName: \"kubernetes.io/projected/830e6413-8bf5-46b8-a643-08f403366a81-kube-api-access-vg72m\") pod \"maas-api-revoke-sub-7fd631b6-9zkrb\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.615062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.615039    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.650172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.650042    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.650779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.650655    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.651107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.651085    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.651441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.651420    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.651941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.651920    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.652109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.652091    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.652480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.652460    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrpf\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-kube-api-access-rsrpf\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.652801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.652651    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.652968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.652950    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f6b01-10e6-4f92-a40f-87f2d6154725-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.677923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.677840    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"]
Oct 30 21:16:30.702852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.702808    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:30.755806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.755044    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.766636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.766593    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.767062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.767036    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.767661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.767641    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.768345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.768325    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.768877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.768860    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.769400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.769378    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrpf\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-kube-api-access-rsrpf\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.770026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.770002    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.770869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.770849    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f6b01-10e6-4f92-a40f-87f2d6154725-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.772360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.772340    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.772621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.767520    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.773164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.769311    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.773283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.771433    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.773370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.772043    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f6b01-10e6-4f92-a40f-87f2d6154725-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.773450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.768209    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.776974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.776932    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.783115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.783091    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrpf\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-kube-api-access-rsrpf\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.783358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.783331    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:30.983051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:30.983015    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:31.017904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.017864    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"]
Oct 30 21:16:31.125073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.125035    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb" event={"ID":"830e6413-8bf5-46b8-a643-08f403366a81","Type":"ContainerStarted","Data":"ccf90c712aa1d5febde1f4324b1dd14cc064520e3f67344d3eeee09dcf0c558e"}
Oct 30 21:16:31.293968 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:16:31.293933    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07f6b01_10e6_4f92_a40f_87f2d6154725.slice/crio-487378a02c27ce38e969ac21e9b4ce6d8ca539d962424a079103a4adf0b03e6b WatchSource:0}: Error finding container 487378a02c27ce38e969ac21e9b4ce6d8ca539d962424a079103a4adf0b03e6b: Status 404 returned error can't find the container with id 487378a02c27ce38e969ac21e9b4ce6d8ca539d962424a079103a4adf0b03e6b
Oct 30 21:16:31.294151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.294130    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"]
Oct 30 21:16:31.296743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.296710    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:31.296835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.296777    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:31.296835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.296808    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:31.936502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.936473    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:16:31.940313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.940286    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:16:31.963696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.962073    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:16:31.966596 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:31.966239    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:16:32.148280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:32.148245    4633 generic.go:356] "Generic (PLEG): container finished" podID="830e6413-8bf5-46b8-a643-08f403366a81" containerID="783e9f2384cec430445be0bf8800dbad26c29380d914b8ac604a183f0823d700" exitCode=0
Oct 30 21:16:32.148754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:32.148342    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb" event={"ID":"830e6413-8bf5-46b8-a643-08f403366a81","Type":"ContainerDied","Data":"783e9f2384cec430445be0bf8800dbad26c29380d914b8ac604a183f0823d700"}
Oct 30 21:16:32.150214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:32.150188    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" event={"ID":"a07f6b01-10e6-4f92-a40f-87f2d6154725","Type":"ContainerStarted","Data":"a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f"}
Oct 30 21:16:32.150320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:32.150232    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" event={"ID":"a07f6b01-10e6-4f92-a40f-87f2d6154725","Type":"ContainerStarted","Data":"487378a02c27ce38e969ac21e9b4ce6d8ca539d962424a079103a4adf0b03e6b"}
Oct 30 21:16:32.183767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:32.183711    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podStartSLOduration=2.183692538 podStartE2EDuration="2.183692538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:32.181833129 +0000 UTC m=+900.531560538" watchObservedRunningTime="2026-10-30 21:16:32.183692538 +0000 UTC m=+900.533419937"
Oct 30 21:16:32.983538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:32.983501    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:32.985105 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:32.985074    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:16:32.985211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:32.985124    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:16:33.451789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.451763    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:33.459529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.459504    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-xks-ca\") pod \"830e6413-8bf5-46b8-a643-08f403366a81\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") "
Oct 30 21:16:33.459651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.459575    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-openshift-ca\") pod \"830e6413-8bf5-46b8-a643-08f403366a81\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") "
Oct 30 21:16:33.459724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.459703    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830e6413-8bf5-46b8-a643-08f403366a81-kube-api-access-vg72m\" (UniqueName: \"kubernetes.io/projected/830e6413-8bf5-46b8-a643-08f403366a81-kube-api-access-vg72m\") pod \"830e6413-8bf5-46b8-a643-08f403366a81\" (UID: \"830e6413-8bf5-46b8-a643-08f403366a81\") "
Oct 30 21:16:33.459932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.459908    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-openshift-ca" pod "830e6413-8bf5-46b8-a643-08f403366a81" (UID: "830e6413-8bf5-46b8-a643-08f403366a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:33.460201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.460186    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-openshift-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.461857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.461834    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-xks-ca" pod "830e6413-8bf5-46b8-a643-08f403366a81" (UID: "830e6413-8bf5-46b8-a643-08f403366a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:33.461949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.461889    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830e6413-8bf5-46b8-a643-08f403366a81-kube-api-access-vg72m" pod "830e6413-8bf5-46b8-a643-08f403366a81" (UID: "830e6413-8bf5-46b8-a643-08f403366a81"). InnerVolumeSpecName "kube-api-access-vg72m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:33.561254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.561170    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830e6413-8bf5-46b8-a643-08f403366a81-maas-api-xks-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.561254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.561202    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg72m\" (UniqueName: \"kubernetes.io/projected/830e6413-8bf5-46b8-a643-08f403366a81-kube-api-access-vg72m\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.983942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.983906    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:16:33.984126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:33.983978    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:16:34.167056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:34.167030    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"
Oct 30 21:16:34.167216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:34.167067    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb" event={"ID":"830e6413-8bf5-46b8-a643-08f403366a81","Type":"ContainerDied","Data":"ccf90c712aa1d5febde1f4324b1dd14cc064520e3f67344d3eeee09dcf0c558e"}
Oct 30 21:16:34.167216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:34.167115    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf90c712aa1d5febde1f4324b1dd14cc064520e3f67344d3eeee09dcf0c558e"
Oct 30 21:16:34.983586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:34.983548    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:16:34.984116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:34.983614    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:16:35.983925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:35.983893    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:16:35.984389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:35.983954    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:16:36.983424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:36.983383    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:16:36.983611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:36.983454    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:16:37.983826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:37.983770    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:16:37.984322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:37.983846    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:16:39.172736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:39.172696    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:39.173227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:39.173144    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:39.173975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:39.173952    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:16:41.335114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.335078    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"]
Oct 30 21:16:41.336087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.336063    4633 cpu_manager.go:414] "containerMap: removing container" podUID="830e6413-8bf5-46b8-a643-08f403366a81" containerName="revoke-keys"
Oct 30 21:16:41.336087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.336086    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830e6413-8bf5-46b8-a643-08f403366a81" containerName="revoke-keys"
Oct 30 21:16:41.337560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.337533    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="830e6413-8bf5-46b8-a643-08f403366a81" containerName="revoke-keys"
Oct 30 21:16:41.344411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.344390    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.354756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.354722    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5bc8c60a-openshift-default-dockercfg-nnv8f\""
Oct 30 21:16:41.361200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.361172    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"]
Oct 30 21:16:41.417517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.417478    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-podinfo\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.417763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.417544    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526p9\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-kube-api-access-526p9\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.417763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.417656    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-token\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.417763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.417717    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-data\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.417934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.417764    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-certs\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.417934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.417812    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.418037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.417948    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-credential-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.418037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.417997    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd236d7-7f72-49a1-850c-642a23a4ef34-istiod-ca-cert\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.418141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.418045    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-envoy\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522469    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-credential-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522552    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd236d7-7f72-49a1-850c-642a23a4ef34-istiod-ca-cert\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522605    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-envoy\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522683    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-podinfo\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522726    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526p9\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-kube-api-access-526p9\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522842    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-token\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522875    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-data\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522909    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-certs\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.522971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.522944    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.523509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.523474    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.523787 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.523761    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-credential-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.524472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.524449    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd236d7-7f72-49a1-850c-642a23a4ef34-istiod-ca-cert\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.527530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.527489    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-envoy\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.567697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.545172    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-data\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.567951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.545840    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-podinfo\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.568075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.546054    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-certs\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.590609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.590526    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-526p9\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-kube-api-access-526p9\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.598025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.597947    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-token\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:41.671115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:41.671022    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:42.006384 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:16:42.006351    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd236d7_7f72_49a1_850c_642a23a4ef34.slice/crio-3473f487588dd563a709f9188ac05069d528b7970b9ee4564de4399dad335d9a WatchSource:0}: Error finding container 3473f487588dd563a709f9188ac05069d528b7970b9ee4564de4399dad335d9a: Status 404 returned error can't find the container with id 3473f487588dd563a709f9188ac05069d528b7970b9ee4564de4399dad335d9a
Oct 30 21:16:42.007570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.007547    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"]
Oct 30 21:16:42.008927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.008896    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:42.009002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.008956    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:42.009002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.008986    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:42.231764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.231724    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" event={"ID":"0cd236d7-7f72-49a1-850c-642a23a4ef34","Type":"ContainerStarted","Data":"4a2950a4835efb8ddf42a5c2cd4a5f489074b719429ed6bc4e4e5912e1b0f2d5"}
Oct 30 21:16:42.231987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.231825    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" event={"ID":"0cd236d7-7f72-49a1-850c-642a23a4ef34","Type":"ContainerStarted","Data":"3473f487588dd563a709f9188ac05069d528b7970b9ee4564de4399dad335d9a"}
Oct 30 21:16:42.257117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.257001    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podStartSLOduration=1.256981011 podStartE2EDuration="1.256981011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:42.252721418 +0000 UTC m=+910.602448815" watchObservedRunningTime="2026-10-30 21:16:42.256981011 +0000 UTC m=+910.606708408"
Oct 30 21:16:42.272819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.272781    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:16:42.671447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:42.671415    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:43.781742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:43.781011    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:44.270697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:44.267101    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:44.292697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:44.276940    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:16:44.739328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:44.739295    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"]
Oct 30 21:16:45.239819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.229085    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.340730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.338903    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6a24e1-openshift-default-dockercfg-tj2wt\""
Oct 30 21:16:45.420700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.414144    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-socket\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.533982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.533873    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-certs\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.537900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.537843    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istiod-ca-cert\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.538362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.538336    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdfs\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-kube-api-access-dtdfs\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.538577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.538546    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-credential-socket\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.538749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.497772    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"]
Oct 30 21:16:45.539426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.539399    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-envoy\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.539547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.539524    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-podinfo\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.539636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.539598    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-token\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.539831 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.539811    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-data\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.643548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.643503    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdfs\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-kube-api-access-dtdfs\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.643916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.643895    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-credential-socket\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.644082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.644065    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-envoy\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.644263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.644248    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-podinfo\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.644397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.644381    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-token\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.644577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.644559    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-data\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.685695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.663613    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-envoy\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.686204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.684136    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-credential-socket\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.686407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.684693    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-podinfo\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.688960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.686999    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-socket\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.688960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.687045    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-certs\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.688960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.687091    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istiod-ca-cert\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.688960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.688100    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-socket\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.688960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.688322    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-certs\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.688960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.688918    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istiod-ca-cert\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.689559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.689539    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-data\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:45.723696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.718272    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"]
Oct 30 21:16:45.735733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:45.730460    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtdfs\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-kube-api-access-dtdfs\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:46.134580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:46.134546    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-token\") pod \"e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") " pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:46.137362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:46.137333    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:46.268602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:46.268120    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.023057    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"]
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.026749    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086368    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086467    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086527    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnmx4\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-kube-api-access-nnmx4\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086571    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086662    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086743    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5182b128-53a7-48f4-915e-f808a22318db-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086792    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5182b128-53a7-48f4-915e-f808a22318db-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086844    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.090699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.086874    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.198816    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.198932    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.198996    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnmx4\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-kube-api-access-nnmx4\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.199036    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.199111    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.199170    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5182b128-53a7-48f4-915e-f808a22318db-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.199222    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5182b128-53a7-48f4-915e-f808a22318db-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.199272    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.247769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.199303    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.258110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.258062    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.321711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.315799    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.348734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.346510    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.348734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.347745    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.348734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.348627    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.446695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.442797    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnmx4\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-kube-api-access-nnmx4\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.518414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.518328    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5182b128-53a7-48f4-915e-f808a22318db-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.523002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.522372    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5182b128-53a7-48f4-915e-f808a22318db-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.616522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.616483    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:16:47.628777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.628740    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.679235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.679198    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.741721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.730891    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:47.904761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:47.890506    4633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded"
Oct 30 21:16:48.323847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:48.323799    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:49.270889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:49.235197    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:49.270889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:49.235271    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:49.348873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:49.334842    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:16:49.898876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:49.868802    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:49.898876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:49.868887    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:50.390145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:50.366158    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:50.874753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:50.873777    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:50.874753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:50.873954    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:50.874753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:50.874071    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:51.339697 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:16:51.278895    4633 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.494s"
Oct 30 21:16:51.530317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:51.464772    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:51.572896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:51.569291    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"]
Oct 30 21:16:51.646105 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:51.646056    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" event={"ID":"c4739bd0-7a3e-40e8-bc58-7665486fbce3","Type":"ContainerStarted","Data":"23f601f4db75ce1dbb6200d879e6621d889278ba97c87db22d12127fe33bb0fb"}
Oct 30 21:16:51.836560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:51.836469    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"]
Oct 30 21:16:52.099767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:52.063952    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:52.118133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:52.118085    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:52.161401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:52.161355    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:16:52.531833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:52.531768    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:16:52.969253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:52.969209    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" event={"ID":"5182b128-53a7-48f4-915e-f808a22318db","Type":"ContainerStarted","Data":"a893503fb4d9fee5ff64dd8e783a44be7c8378dd811ce47ed73336ae5ec81142"}
Oct 30 21:16:53.106405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:53.106364    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" event={"ID":"c4739bd0-7a3e-40e8-bc58-7665486fbce3","Type":"ContainerStarted","Data":"156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba"}
Oct 30 21:16:53.139169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:53.139131    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:53.159779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:53.159726    4633 patch_prober.go:29] interesting pod/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:16:53.160086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:53.160042    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 21:16:53.219689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:53.219533    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podStartSLOduration=9.219511895 podStartE2EDuration="9.219511895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:53.179230762 +0000 UTC m=+921.528958158" watchObservedRunningTime="2026-10-30 21:16:53.219511895 +0000 UTC m=+921.569239292"
Oct 30 21:16:54.144694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:54.144598    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" event={"ID":"5182b128-53a7-48f4-915e-f808a22318db","Type":"ContainerStarted","Data":"d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e"}
Oct 30 21:16:54.152993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:54.152965    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:54.216693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:54.214241    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podStartSLOduration=9.214221507 podStartE2EDuration="9.214221507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:54.202373371 +0000 UTC m=+922.552100767" watchObservedRunningTime="2026-10-30 21:16:54.214221507 +0000 UTC m=+922.563948904"
Oct 30 21:16:54.768945 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:54.768908    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:16:54.769392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:54.769367    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:16:54.769581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:54.769557    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 21:16:55.045897 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:16:55.045803    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:16:55.150809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:55.150777    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:55.153263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:55.153237    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:16:55.749058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:55.749022    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:16:55.749245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:55.749081    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 21:16:56.748654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:56.748612    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:16:56.748654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:56.748680    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 21:16:57.048182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.048052    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"]
Oct 30 21:16:57.053948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.053920    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"]
Oct 30 21:16:57.054126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.054033    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:57.070164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.070135    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6a24e1\""
Oct 30 21:16:57.070479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.070461    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6a24e1-dockercfg-bn4jq\""
Oct 30 21:16:57.093763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.090623    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsz2s\" (UniqueName: \"kubernetes.io/projected/7f478165-d481-4adc-9113-4c3cbcbb69b6-kube-api-access-gsz2s\") pod \"payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c\" (UID: \"7f478165-d481-4adc-9113-4c3cbcbb69b6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:57.093763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.090776    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f478165-d481-4adc-9113-4c3cbcbb69b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c\" (UID: \"7f478165-d481-4adc-9113-4c3cbcbb69b6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:57.105955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.105924    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"]
Oct 30 21:16:57.112206 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.112165    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:57.156607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.156576    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"]
Oct 30 21:16:57.199716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.198377    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz2s\" (UniqueName: \"kubernetes.io/projected/7f478165-d481-4adc-9113-4c3cbcbb69b6-kube-api-access-gsz2s\") pod \"payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c\" (UID: \"7f478165-d481-4adc-9113-4c3cbcbb69b6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:57.199716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.198581    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f478165-d481-4adc-9113-4c3cbcbb69b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c\" (UID: \"7f478165-d481-4adc-9113-4c3cbcbb69b6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:57.264708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.237931    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f478165-d481-4adc-9113-4c3cbcbb69b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c\" (UID: \"7f478165-d481-4adc-9113-4c3cbcbb69b6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:57.386750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.384611    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz2s\" (UniqueName: \"kubernetes.io/projected/7f478165-d481-4adc-9113-4c3cbcbb69b6-kube-api-access-gsz2s\") pod \"payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c\" (UID: \"7f478165-d481-4adc-9113-4c3cbcbb69b6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:57.398386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.398087    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:57.428691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.424148    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1533f3-2ef5-4499-a0df-5bf52daf6610-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm\" (UID: \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:57.428691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.424323    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjzc7\" (UniqueName: \"kubernetes.io/projected/5d1533f3-2ef5-4499-a0df-5bf52daf6610-kube-api-access-bjzc7\") pod \"payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm\" (UID: \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:57.544696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.543743    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1533f3-2ef5-4499-a0df-5bf52daf6610-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm\" (UID: \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:57.544696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.544025    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjzc7\" (UniqueName: \"kubernetes.io/projected/5d1533f3-2ef5-4499-a0df-5bf52daf6610-kube-api-access-bjzc7\") pod \"payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm\" (UID: \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:57.546760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.546562    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1533f3-2ef5-4499-a0df-5bf52daf6610-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm\" (UID: \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:57.553444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.553419    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjzc7\" (UniqueName: \"kubernetes.io/projected/5d1533f3-2ef5-4499-a0df-5bf52daf6610-kube-api-access-bjzc7\") pod \"payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm\" (UID: \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\") " pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:57.748496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.748456    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:16:57.748662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.748519    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 21:16:57.763241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.763214    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:57.860091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:57.860063    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"]
Oct 30 21:16:57.866710 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:16:57.864970    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f478165_d481_4adc_9113_4c3cbcbb69b6.slice/crio-11685002dccda057334693b0416509cea8360214040637ca63b522ec97d33275 WatchSource:0}: Error finding container 11685002dccda057334693b0416509cea8360214040637ca63b522ec97d33275: Status 404 returned error can't find the container with id 11685002dccda057334693b0416509cea8360214040637ca63b522ec97d33275
Oct 30 21:16:58.172744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:58.145632    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"]
Oct 30 21:16:58.249584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:58.249550    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm" event={"ID":"5d1533f3-2ef5-4499-a0df-5bf52daf6610","Type":"ContainerStarted","Data":"5234770e6b8f34df4e69e3368ae13a76943ad3550bc13c1b74c8b1f45e45748d"}
Oct 30 21:16:58.251285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:58.251251    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c" event={"ID":"7f478165-d481-4adc-9113-4c3cbcbb69b6","Type":"ContainerStarted","Data":"4ae2c6d9ad85258e19213518f4e48ec925c0ca9c6c9499754bfe3fccb341b2b2"}
Oct 30 21:16:58.251417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:58.251317    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:16:58.251417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:58.251341    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c" event={"ID":"7f478165-d481-4adc-9113-4c3cbcbb69b6","Type":"ContainerStarted","Data":"11685002dccda057334693b0416509cea8360214040637ca63b522ec97d33275"}
Oct 30 21:16:58.277317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:58.276229    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c" podStartSLOduration=1.276215128 podStartE2EDuration="1.276215128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:58.275178971 +0000 UTC m=+926.624906369" watchObservedRunningTime="2026-10-30 21:16:58.276215128 +0000 UTC m=+926.625942523"
Oct 30 21:16:58.749694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:58.749086    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:16:58.749694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:58.749145    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 21:16:59.261047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.261012    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm" event={"ID":"5d1533f3-2ef5-4499-a0df-5bf52daf6610","Type":"ContainerStarted","Data":"e642d2b857c28154d3b33a9abaf4c468d790ce39fffb5d636f7504f9dd3968c0"}
Oct 30 21:16:59.261503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.261080    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:16:59.315930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.315866    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm" podStartSLOduration=2.315843493 podStartE2EDuration="2.315843493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:59.304313037 +0000 UTC m=+927.654040427" watchObservedRunningTime="2026-10-30 21:16:59.315843493 +0000 UTC m=+927.665570890"
Oct 30 21:16:59.748527 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.748495    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:16:59.748739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.748548    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 21:16:59.764045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.764008    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"]
Oct 30 21:16:59.769405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.769382    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:16:59.778154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.778123    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-86671d-dockercfg-zwhsp\""
Oct 30 21:16:59.798473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.798444    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"]
Oct 30 21:16:59.810733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.810701    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6fr\" (UniqueName: \"kubernetes.io/projected/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-kube-api-access-bm6fr\") pod \"payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm\" (UID: \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:16:59.810876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.810847    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm\" (UID: \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:16:59.874737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.872959    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-86671d\""
Oct 30 21:16:59.934541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.931484    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm\" (UID: \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:16:59.934541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.931829    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6fr\" (UniqueName: \"kubernetes.io/projected/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-kube-api-access-bm6fr\") pod \"payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm\" (UID: \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:16:59.934541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:16:59.933085    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm\" (UID: \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:17:00.137538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:00.126743    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6fr\" (UniqueName: \"kubernetes.io/projected/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-kube-api-access-bm6fr\") pod \"payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm\" (UID: \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:17:00.391954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:00.391863    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:17:00.748470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:00.748434    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:17:00.748628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:00.748488    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 21:17:00.825508 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:17:00.825481    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cbfce7_e9be_44b8_bed5_9cfd10ac992d.slice/crio-774bff1d96cbe74ba163553dbd9f8df308fcc6d5277a5716a03e4491792c01fa WatchSource:0}: Error finding container 774bff1d96cbe74ba163553dbd9f8df308fcc6d5277a5716a03e4491792c01fa: Status 404 returned error can't find the container with id 774bff1d96cbe74ba163553dbd9f8df308fcc6d5277a5716a03e4491792c01fa
Oct 30 21:17:00.826998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:00.826970    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"]
Oct 30 21:17:01.287392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:01.287343    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm" event={"ID":"65cbfce7-e9be-44b8-bed5-9cfd10ac992d","Type":"ContainerStarted","Data":"dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5"}
Oct 30 21:17:01.287572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:01.287408    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm" event={"ID":"65cbfce7-e9be-44b8-bed5-9cfd10ac992d","Type":"ContainerStarted","Data":"774bff1d96cbe74ba163553dbd9f8df308fcc6d5277a5716a03e4491792c01fa"}
Oct 30 21:17:01.287572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:01.287438    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:17:01.306306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:01.306254    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm" podStartSLOduration=2.306241122 podStartE2EDuration="2.306241122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:01.303971256 +0000 UTC m=+929.653698653" watchObservedRunningTime="2026-10-30 21:17:01.306241122 +0000 UTC m=+929.655968584"
Oct 30 21:17:01.932015 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:01.931981    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:17:01.933038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:01.932969    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:17:01.933767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:01.933735    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:17:07.403698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:07.392769    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:07.593879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:07.593814    4633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.385955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:08.385917    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:08.439163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:08.439096    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.801704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:08.777412    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-md8nn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:08.884267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:08.884215    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.024867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:09.024829    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-wlw8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:09.025153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:09.025122    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.052752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:09.051949    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-8tv48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:09.052752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:09.052027    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-8tv48" podUID="cae247f2-d22b-4593-9935-071384bb7be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.392956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:09.392865    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:17:10.419928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:10.419892    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:17:11.547517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.547481    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"]
Oct 30 21:17:11.552814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.552793    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.561652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.561627    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-86671d-serving-cert\""
Oct 30 21:17:11.575077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.575051    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"]
Oct 30 21:17:11.662154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.662115    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvr7\" (UniqueName: \"kubernetes.io/projected/d242aa53-5566-4147-a390-1200dd699848-kube-api-access-hwvr7\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.662329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.662309    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-metrics-certs\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.662411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.662399    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d242aa53-5566-4147-a390-1200dd699848-openshift-service-ca\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.662508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.662497    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-maas-api-tls\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.765998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.763914    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d242aa53-5566-4147-a390-1200dd699848-openshift-service-ca\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.765998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.764092    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-maas-api-tls\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.765998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.764182    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvr7\" (UniqueName: \"kubernetes.io/projected/d242aa53-5566-4147-a390-1200dd699848-kube-api-access-hwvr7\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.765998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.764268    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-metrics-certs\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.768373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.768344    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d242aa53-5566-4147-a390-1200dd699848-openshift-service-ca\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.768582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.768560    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-maas-api-tls\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.769840 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.769811    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-metrics-certs\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.776361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.776331    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvr7\" (UniqueName: \"kubernetes.io/projected/d242aa53-5566-4147-a390-1200dd699848-kube-api-access-hwvr7\") pod \"maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:11.876812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:11.876716    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:12.303898 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:17:12.303866    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd242aa53_5566_4147_a390_1200dd699848.slice/crio-d977209a31ab1ca9b3ae4555d21794db3ea4cc1c365b1c8e4151310f637e1c14 WatchSource:0}: Error finding container d977209a31ab1ca9b3ae4555d21794db3ea4cc1c365b1c8e4151310f637e1c14: Status 404 returned error can't find the container with id d977209a31ab1ca9b3ae4555d21794db3ea4cc1c365b1c8e4151310f637e1c14
Oct 30 21:17:12.304746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:12.304719    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"]
Oct 30 21:17:12.305748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:12.305719    4633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:17:12.308281 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:12.308230    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:17:12.639755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:12.637839    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4" event={"ID":"d242aa53-5566-4147-a390-1200dd699848","Type":"ContainerStarted","Data":"d977209a31ab1ca9b3ae4555d21794db3ea4cc1c365b1c8e4151310f637e1c14"}
Oct 30 21:17:13.647022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:13.646985    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4" event={"ID":"d242aa53-5566-4147-a390-1200dd699848","Type":"ContainerStarted","Data":"92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170"}
Oct 30 21:17:13.647408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:13.647049    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:13.668238 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:13.668193    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4" podStartSLOduration=2.288263093 podStartE2EDuration="2.668179056s" totalImagesPullingTime="379.915963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:13.664560688 +0000 UTC m=+942.014288087" watchObservedRunningTime="2026-10-30 21:17:13.668179056 +0000 UTC m=+942.017906452"
Oct 30 21:17:19.660417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:19.660382    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:17:21.971692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:21.971012    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"]
Oct 30 21:17:21.979739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:21.979709    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:21.983719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:21.983217    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c499bf-openshift-default-dockercfg-9t89f\""
Oct 30 21:17:22.002304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:21.986739    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"]
Oct 30 21:17:22.104326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.104292    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-socket\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.104590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.104575    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-envoy\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.104855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.104837    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-certs\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.105092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.105075    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55172deb-27cf-4d37-8919-dc78904d47f7-istio-podinfo\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.105213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.105199    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-istio-token\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.105343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.105327    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45g6\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-kube-api-access-b45g6\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.105457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.105444    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-data\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.105564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.105552    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55172deb-27cf-4d37-8919-dc78904d47f7-istiod-ca-cert\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.105720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.105705    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-credential-socket\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207246    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-credential-socket\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207333    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-socket\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207407    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-envoy\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207488    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-certs\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207578    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55172deb-27cf-4d37-8919-dc78904d47f7-istio-podinfo\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207606    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-istio-token\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207652    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45g6\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-kube-api-access-b45g6\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207805    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-data\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.207867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.207852    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55172deb-27cf-4d37-8919-dc78904d47f7-istiod-ca-cert\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.208573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.208544    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-credential-socket\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.208745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.208575    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55172deb-27cf-4d37-8919-dc78904d47f7-istiod-ca-cert\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.208745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.208729    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-socket\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.208877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.208841    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-certs\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.208877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.208846    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-data\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.210257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.210228    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-envoy\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.210373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.210337    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55172deb-27cf-4d37-8919-dc78904d47f7-istio-podinfo\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.217292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.217263    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-istio-token\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.218063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.218041    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45g6\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-kube-api-access-b45g6\") pod \"e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") " pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.308745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.305952    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:22.771857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.769023    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"]
Oct 30 21:17:22.774543 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:17:22.774510    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55172deb_27cf_4d37_8919_dc78904d47f7.slice/crio-b5015159b027fe3e6cdcc31994abf2492e06c18cb446944ee2ac8e5bf6cf9ddf WatchSource:0}: Error finding container b5015159b027fe3e6cdcc31994abf2492e06c18cb446944ee2ac8e5bf6cf9ddf: Status 404 returned error can't find the container with id b5015159b027fe3e6cdcc31994abf2492e06c18cb446944ee2ac8e5bf6cf9ddf
Oct 30 21:17:22.777237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.777202    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:22.777334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.777275    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:22.777334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.777308    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:22.940883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:22.940850    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:17:23.738972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:23.738926    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" event={"ID":"55172deb-27cf-4d37-8919-dc78904d47f7","Type":"ContainerStarted","Data":"8b16d78c2e74e24095c682f676aabe0f2d4631cb517f26915bad4e2423413c77"}
Oct 30 21:17:23.739417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:23.738997    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" event={"ID":"55172deb-27cf-4d37-8919-dc78904d47f7","Type":"ContainerStarted","Data":"b5015159b027fe3e6cdcc31994abf2492e06c18cb446944ee2ac8e5bf6cf9ddf"}
Oct 30 21:17:23.767739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:23.767663    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podStartSLOduration=2.767644832 podStartE2EDuration="2.767644832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:23.76312612 +0000 UTC m=+952.112853516" watchObservedRunningTime="2026-10-30 21:17:23.767644832 +0000 UTC m=+952.117372229"
Oct 30 21:17:24.307127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:24.307089    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:24.314105 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:24.314077    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:24.748131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:24.748097    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:24.749105 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:24.749085    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:17:25.050453 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:17:25.050346    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:17:33.765958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:33.765924    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"]
Oct 30 21:17:33.791693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:33.791428    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"]
Oct 30 21:17:33.791693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:33.791507    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:33.803048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:33.799889    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c499bf-serving-cert\""
Oct 30 21:17:33.926453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:33.926422    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-maas-api-tls\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:33.926645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:33.926616    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-metrics-certs\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:33.926795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:33.926771    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldbzr\" (UniqueName: \"kubernetes.io/projected/f6e9614a-bfed-4d4d-97f3-41b5713da79d-kube-api-access-ldbzr\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:33.926874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:33.926857    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9614a-bfed-4d4d-97f3-41b5713da79d-openshift-service-ca\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.032879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.032773    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-maas-api-tls\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.047454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.046242    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-metrics-certs\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.047454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.046365    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldbzr\" (UniqueName: \"kubernetes.io/projected/f6e9614a-bfed-4d4d-97f3-41b5713da79d-kube-api-access-ldbzr\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.047454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.046437    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9614a-bfed-4d4d-97f3-41b5713da79d-openshift-service-ca\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.047454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.047398    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9614a-bfed-4d4d-97f3-41b5713da79d-openshift-service-ca\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.092741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.091944    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-maas-api-tls\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.097371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.097298    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldbzr\" (UniqueName: \"kubernetes.io/projected/f6e9614a-bfed-4d4d-97f3-41b5713da79d-kube-api-access-ldbzr\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.108362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.108332    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-metrics-certs\") pod \"maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.130488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.130458    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:34.482704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.482386    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"]
Oct 30 21:17:34.490364 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:17:34.490309    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e9614a_bfed_4d4d_97f3_41b5713da79d.slice/crio-a7e2c81007eec8faa254655631df45076e3be5c2f7db6136ea630e6be7b0f318 WatchSource:0}: Error finding container a7e2c81007eec8faa254655631df45076e3be5c2f7db6136ea630e6be7b0f318: Status 404 returned error can't find the container with id a7e2c81007eec8faa254655631df45076e3be5c2f7db6136ea630e6be7b0f318
Oct 30 21:17:34.964101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:34.964067    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8" event={"ID":"f6e9614a-bfed-4d4d-97f3-41b5713da79d","Type":"ContainerStarted","Data":"a7e2c81007eec8faa254655631df45076e3be5c2f7db6136ea630e6be7b0f318"}
Oct 30 21:17:35.974993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:35.974956    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8" event={"ID":"f6e9614a-bfed-4d4d-97f3-41b5713da79d","Type":"ContainerStarted","Data":"dfdd96291c76e5aa36b2a43278ee7f4f938b3ded431f182cb84a7624071fe9ec"}
Oct 30 21:17:35.975422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:35.975093    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:36.045691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:36.045161    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8" podStartSLOduration=2.645466332 podStartE2EDuration="3.045138738s" totalImagesPullingTime="399.672406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:36.012205352 +0000 UTC m=+964.361932748" watchObservedRunningTime="2026-10-30 21:17:36.045138738 +0000 UTC m=+964.394866153"
Oct 30 21:17:36.983652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:36.983619    4633 generic.go:356] "Generic (PLEG): container finished" podID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerID="156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba" exitCode=0
Oct 30 21:17:36.984207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:36.983699    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" event={"ID":"c4739bd0-7a3e-40e8-bc58-7665486fbce3","Type":"ContainerDied","Data":"156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba"}
Oct 30 21:17:36.984302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:36.984280    4633 scope.go:122] "RemoveContainer" containerID="156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba"
Oct 30 21:17:36.985585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:36.985560    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:36.985646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:36.985626    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:36.985717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:36.985655    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:37.993740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:37.993701    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" event={"ID":"c4739bd0-7a3e-40e8-bc58-7665486fbce3","Type":"ContainerStarted","Data":"c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c"}
Oct 30 21:17:38.139056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:38.139012    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:17:38.140798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:38.140770    4633 patch_prober.go:29] interesting pod/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:17:38.140942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:38.140823    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 21:17:39.138310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:39.138262    4633 patch_prober.go:29] interesting pod/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:17:39.138898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:39.138349    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 21:17:40.144309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:40.144265    4633 patch_prober.go:29] interesting pod/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:17:40.144968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:40.144353    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 21:17:40.152653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:40.152617    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:17:41.154836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:41.154788    4633 patch_prober.go:29] interesting pod/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:17:41.155480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:41.155444    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 21:17:42.007697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:42.007134    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:17:42.178696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:42.170394    4633 patch_prober.go:29] interesting pod/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:17:42.178696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:42.170454    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 21:17:43.141695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:43.141267    4633 patch_prober.go:29] interesting pod/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 21:17:43.141695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:43.141333    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 21:17:44.144880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:44.144847    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:17:44.146112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:44.146094    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:17:46.661971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.661935    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"]
Oct 30 21:17:46.666339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.666321    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.672107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.672081    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b3d1de-serving-cert\""
Oct 30 21:17:46.685970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.685925    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"]
Oct 30 21:17:46.795481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.795439    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6844674a-d6eb-4b58-94db-c77eec954d6f-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.795691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.795496    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8vh\" (UniqueName: \"kubernetes.io/projected/6844674a-d6eb-4b58-94db-c77eec954d6f-kube-api-access-dj8vh\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.795751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.795688    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-maas-api-tls\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.795935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.795913    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-metrics-certs\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.897313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.897265    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-metrics-certs\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.897555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.897529    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6844674a-d6eb-4b58-94db-c77eec954d6f-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.897645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.897574    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8vh\" (UniqueName: \"kubernetes.io/projected/6844674a-d6eb-4b58-94db-c77eec954d6f-kube-api-access-dj8vh\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.897645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.897638    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-maas-api-tls\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.898440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.898407    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6844674a-d6eb-4b58-94db-c77eec954d6f-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.930621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.900230    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-metrics-certs\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.930621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.900313    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-maas-api-tls\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.950695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.935123    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8vh\" (UniqueName: \"kubernetes.io/projected/6844674a-d6eb-4b58-94db-c77eec954d6f-kube-api-access-dj8vh\") pod \"maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:46.999781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:46.999751    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:47.362163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:47.362007    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"]
Oct 30 21:17:47.405571 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:47.405534    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr" event={"ID":"6844674a-d6eb-4b58-94db-c77eec954d6f","Type":"ContainerStarted","Data":"80f377d97726297d1ca92ad5021e5efb0758d3d42dc9ec532c831cbd7a1c1db3"}
Oct 30 21:17:48.427628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:48.427584    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr" event={"ID":"6844674a-d6eb-4b58-94db-c77eec954d6f","Type":"ContainerStarted","Data":"8e76caa43c7ae0ec4db876575371ef903eb02e024431b92879e9c29bbcf601bd"}
Oct 30 21:17:48.428156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:48.427686    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:48.447976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:48.447924    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr" podStartSLOduration=1.910644234 podStartE2EDuration="2.447910996s" totalImagesPullingTime="537.266762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:48.445191345 +0000 UTC m=+976.794918742" watchObservedRunningTime="2026-10-30 21:17:48.447910996 +0000 UTC m=+976.797638429"
Oct 30 21:17:51.490080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:51.490044    4633 generic.go:356] "Generic (PLEG): container finished" podID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerID="4a2950a4835efb8ddf42a5c2cd4a5f489074b719429ed6bc4e4e5912e1b0f2d5" exitCode=0
Oct 30 21:17:51.490080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:51.490093    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" event={"ID":"0cd236d7-7f72-49a1-850c-642a23a4ef34","Type":"ContainerDied","Data":"4a2950a4835efb8ddf42a5c2cd4a5f489074b719429ed6bc4e4e5912e1b0f2d5"}
Oct 30 21:17:51.490653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:51.490557    4633 scope.go:122] "RemoveContainer" containerID="4a2950a4835efb8ddf42a5c2cd4a5f489074b719429ed6bc4e4e5912e1b0f2d5"
Oct 30 21:17:51.492136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:51.492106    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:51.492199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:51.492173    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:51.492235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:51.492204    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:17:52.572324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:52.572281    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" event={"ID":"0cd236d7-7f72-49a1-850c-642a23a4ef34","Type":"ContainerStarted","Data":"3bb1dd6dc74bb4311b3cc1f6c82089a412b1c62696ef554422d7b937ac797322"}
Oct 30 21:17:52.671734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:52.671704    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:17:52.671939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:52.671918    4633 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:17:52.671989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:52.671963    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 21:17:53.672279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:53.672243    4633 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:17:53.672797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:53.672299    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 21:17:54.444011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:54.443978    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:17:54.686272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:54.681764    4633 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:17:54.686272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:54.681836    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 21:17:55.043171 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:17:55.043130    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:17:55.672332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:55.672292    4633 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:17:55.672516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:55.672355    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 21:17:56.672371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:56.672337    4633 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:17:56.672799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:56.672389    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 21:17:57.083463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:57.083382    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:17:57.671650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:57.671613    4633 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:17:57.671987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:57.671956    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 21:17:58.571327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:58.571284    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"]
Oct 30 21:17:58.580653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:58.580603    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"]
Oct 30 21:17:58.681882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:58.681839    4633 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Oct 30 21:17:58.682081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:58.681920    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Oct 30 21:17:59.267258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:17:59.267215    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:18:00.183145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:00.183110    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:18:00.184870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:00.184845    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:18:03.572620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:03.572564    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm" podUID="5d1533f3-2ef5-4499-a0df-5bf52daf6610" containerName="payload-processing" containerID="cri-o://e642d2b857c28154d3b33a9abaf4c468d790ce39fffb5d636f7504f9dd3968c0" gracePeriod=25
Oct 30 21:18:03.582171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:03.582124    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c" podUID="7f478165-d481-4adc-9113-4c3cbcbb69b6" containerName="payload-pre-processing" containerID="cri-o://4ae2c6d9ad85258e19213518f4e48ec925c0ca9c6c9499754bfe3fccb341b2b2" gracePeriod=25
Oct 30 21:18:03.810717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:03.809758    4633 generic.go:356] "Generic (PLEG): container finished" podID="5d1533f3-2ef5-4499-a0df-5bf52daf6610" containerID="e642d2b857c28154d3b33a9abaf4c468d790ce39fffb5d636f7504f9dd3968c0" exitCode=0
Oct 30 21:18:03.810717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:03.809831    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm" event={"ID":"5d1533f3-2ef5-4499-a0df-5bf52daf6610","Type":"ContainerDied","Data":"e642d2b857c28154d3b33a9abaf4c468d790ce39fffb5d636f7504f9dd3968c0"}
Oct 30 21:18:03.811953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:03.811552    4633 generic.go:356] "Generic (PLEG): container finished" podID="7f478165-d481-4adc-9113-4c3cbcbb69b6" containerID="4ae2c6d9ad85258e19213518f4e48ec925c0ca9c6c9499754bfe3fccb341b2b2" exitCode=0
Oct 30 21:18:03.811953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:03.811633    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c" event={"ID":"7f478165-d481-4adc-9113-4c3cbcbb69b6","Type":"ContainerDied","Data":"4ae2c6d9ad85258e19213518f4e48ec925c0ca9c6c9499754bfe3fccb341b2b2"}
Oct 30 21:18:03.968231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:03.968200    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:18:03.971978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:03.971852    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:18:04.164690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.157415    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1533f3-2ef5-4499-a0df-5bf52daf6610-plugins-config-volume" pod "5d1533f3-2ef5-4499-a0df-5bf52daf6610" (UID: "5d1533f3-2ef5-4499-a0df-5bf52daf6610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:04.164690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.157502    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1533f3-2ef5-4499-a0df-5bf52daf6610-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1533f3-2ef5-4499-a0df-5bf52daf6610-plugins-config-volume\") pod \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\" (UID: \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\") "
Oct 30 21:18:04.164690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.158948    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f478165-d481-4adc-9113-4c3cbcbb69b6-kube-api-access-gsz2s\" (UniqueName: \"kubernetes.io/projected/7f478165-d481-4adc-9113-4c3cbcbb69b6-kube-api-access-gsz2s\") pod \"7f478165-d481-4adc-9113-4c3cbcbb69b6\" (UID: \"7f478165-d481-4adc-9113-4c3cbcbb69b6\") "
Oct 30 21:18:04.180626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.176612    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1533f3-2ef5-4499-a0df-5bf52daf6610-kube-api-access-bjzc7\" (UniqueName: \"kubernetes.io/projected/5d1533f3-2ef5-4499-a0df-5bf52daf6610-kube-api-access-bjzc7\") pod \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\" (UID: \"5d1533f3-2ef5-4499-a0df-5bf52daf6610\") "
Oct 30 21:18:04.180626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.177432    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f478165-d481-4adc-9113-4c3cbcbb69b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f478165-d481-4adc-9113-4c3cbcbb69b6-plugins-config-volume\") pod \"7f478165-d481-4adc-9113-4c3cbcbb69b6\" (UID: \"7f478165-d481-4adc-9113-4c3cbcbb69b6\") "
Oct 30 21:18:04.180626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.177906    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f478165-d481-4adc-9113-4c3cbcbb69b6-plugins-config-volume" pod "7f478165-d481-4adc-9113-4c3cbcbb69b6" (UID: "7f478165-d481-4adc-9113-4c3cbcbb69b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:04.180626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.179049    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1533f3-2ef5-4499-a0df-5bf52daf6610-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:04.180626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.179073    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f478165-d481-4adc-9113-4c3cbcbb69b6-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:04.245711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.244962    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1533f3-2ef5-4499-a0df-5bf52daf6610-kube-api-access-bjzc7" pod "5d1533f3-2ef5-4499-a0df-5bf52daf6610" (UID: "5d1533f3-2ef5-4499-a0df-5bf52daf6610"). InnerVolumeSpecName "kube-api-access-bjzc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:04.246991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.246942    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f478165-d481-4adc-9113-4c3cbcbb69b6-kube-api-access-gsz2s" pod "7f478165-d481-4adc-9113-4c3cbcbb69b6" (UID: "7f478165-d481-4adc-9113-4c3cbcbb69b6"). InnerVolumeSpecName "kube-api-access-gsz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:04.280345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.280312    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsz2s\" (UniqueName: \"kubernetes.io/projected/7f478165-d481-4adc-9113-4c3cbcbb69b6-kube-api-access-gsz2s\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:04.280345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.280341    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjzc7\" (UniqueName: \"kubernetes.io/projected/5d1533f3-2ef5-4499-a0df-5bf52daf6610-kube-api-access-bjzc7\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:04.822597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.822498    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm" event={"ID":"5d1533f3-2ef5-4499-a0df-5bf52daf6610","Type":"ContainerDied","Data":"5234770e6b8f34df4e69e3368ae13a76943ad3550bc13c1b74c8b1f45e45748d"}
Oct 30 21:18:04.822597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.822548    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"
Oct 30 21:18:04.822597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.822582    4633 scope.go:122] "RemoveContainer" containerID="e642d2b857c28154d3b33a9abaf4c468d790ce39fffb5d636f7504f9dd3968c0"
Oct 30 21:18:04.824497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.824477    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"
Oct 30 21:18:04.824635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.824604    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c" event={"ID":"7f478165-d481-4adc-9113-4c3cbcbb69b6","Type":"ContainerDied","Data":"11685002dccda057334693b0416509cea8360214040637ca63b522ec97d33275"}
Oct 30 21:18:04.842924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.842898    4633 scope.go:122] "RemoveContainer" containerID="4ae2c6d9ad85258e19213518f4e48ec925c0ca9c6c9499754bfe3fccb341b2b2"
Oct 30 21:18:04.860017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.859992    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"]
Oct 30 21:18:04.864563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.864537    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a24e1-76df8d889f-zhs9c"]
Oct 30 21:18:04.870422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.870391    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"]
Oct 30 21:18:04.884838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:04.884802    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6a24e1-cdccdc565-jmwhm"]
Oct 30 21:18:05.796046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:05.796011    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1533f3-2ef5-4499-a0df-5bf52daf6610" path="/var/lib/kubelet/pods/5d1533f3-2ef5-4499-a0df-5bf52daf6610/volumes"
Oct 30 21:18:05.796652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:05.796623    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f478165-d481-4adc-9113-4c3cbcbb69b6" path="/var/lib/kubelet/pods/7f478165-d481-4adc-9113-4c3cbcbb69b6/volumes"
Oct 30 21:18:09.750076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:09.750044    4633 patch_prober.go:29] interesting pod/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:18:09.750495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:09.750113    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:18:09.881220 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:09.881189    4633 generic.go:356] "Generic (PLEG): container finished" podID="55172deb-27cf-4d37-8919-dc78904d47f7" containerID="8b16d78c2e74e24095c682f676aabe0f2d4631cb517f26915bad4e2423413c77" exitCode=0
Oct 30 21:18:09.881380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:09.881266    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" event={"ID":"55172deb-27cf-4d37-8919-dc78904d47f7","Type":"ContainerDied","Data":"8b16d78c2e74e24095c682f676aabe0f2d4631cb517f26915bad4e2423413c77"}
Oct 30 21:18:09.881737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:09.881723    4633 scope.go:122] "RemoveContainer" containerID="8b16d78c2e74e24095c682f676aabe0f2d4631cb517f26915bad4e2423413c77"
Oct 30 21:18:09.883386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:09.883360    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:09.883447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:09.883433    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:09.883482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:09.883466    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:10.892074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:10.892037    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" event={"ID":"55172deb-27cf-4d37-8919-dc78904d47f7","Type":"ContainerStarted","Data":"6614994fd3b3f3c94d301472471a3c426bcf7c031f02209cdb91bbac98b7cf14"}
Oct 30 21:18:11.198490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:11.198456    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"]
Oct 30 21:18:11.306852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:11.306818    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:18:11.307097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:11.306970    4633 patch_prober.go:29] interesting pod/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:18:11.307097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:11.307019    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:18:12.307287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:12.307253    4633 patch_prober.go:29] interesting pod/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:18:12.307704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:12.307306    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:18:13.307569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:13.307532    4633 patch_prober.go:29] interesting pod/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:18:13.308023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:13.307590    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:18:14.307555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:14.307519    4633 patch_prober.go:29] interesting pod/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:18:14.307773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:14.307586    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:18:14.466873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:14.466832    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"]
Oct 30 21:18:14.467259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:14.467227    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy" containerID="cri-o://c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c" gracePeriod=30
Oct 30 21:18:15.307018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:15.306981    4633 patch_prober.go:29] interesting pod/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:18:15.307200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:15.307038    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:18:16.199364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.199294    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm" podUID="65cbfce7-e9be-44b8-bed5-9cfd10ac992d" containerName="payload-pre-processing" containerID="cri-o://dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5" gracePeriod=25
Oct 30 21:18:16.307465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.307425    4633 patch_prober.go:29] interesting pod/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:18:16.307628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.307504    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 21:18:16.589847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.589814    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:18:16.690347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.690310    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-kube-api-access-bm6fr\" (UniqueName: \"kubernetes.io/projected/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-kube-api-access-bm6fr\") pod \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\" (UID: \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\") "
Oct 30 21:18:16.690522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.690361    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-plugins-config-volume\") pod \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\" (UID: \"65cbfce7-e9be-44b8-bed5-9cfd10ac992d\") "
Oct 30 21:18:16.690835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.690808    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-plugins-config-volume" pod "65cbfce7-e9be-44b8-bed5-9cfd10ac992d" (UID: "65cbfce7-e9be-44b8-bed5-9cfd10ac992d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:16.690961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.690949    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:16.692912 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.692888    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-kube-api-access-bm6fr" pod "65cbfce7-e9be-44b8-bed5-9cfd10ac992d" (UID: "65cbfce7-e9be-44b8-bed5-9cfd10ac992d"). InnerVolumeSpecName "kube-api-access-bm6fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:16.792756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:16.792721    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6fr\" (UniqueName: \"kubernetes.io/projected/65cbfce7-e9be-44b8-bed5-9cfd10ac992d-kube-api-access-bm6fr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:17.045716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.045648    4633 generic.go:356] "Generic (PLEG): container finished" podID="65cbfce7-e9be-44b8-bed5-9cfd10ac992d" containerID="dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5" exitCode=0
Oct 30 21:18:17.045716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.045702    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm" event={"ID":"65cbfce7-e9be-44b8-bed5-9cfd10ac992d","Type":"ContainerDied","Data":"dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5"}
Oct 30 21:18:17.045953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.045773    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm" event={"ID":"65cbfce7-e9be-44b8-bed5-9cfd10ac992d","Type":"ContainerDied","Data":"774bff1d96cbe74ba163553dbd9f8df308fcc6d5277a5716a03e4491792c01fa"}
Oct 30 21:18:17.045953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.045797    4633 scope.go:122] "RemoveContainer" containerID="dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5"
Oct 30 21:18:17.045953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.045716    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"
Oct 30 21:18:17.058017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.057971    4633 scope.go:122] "RemoveContainer" containerID="dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5"
Oct 30 21:18:17.058378 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:18:17.058356    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5\": container with ID starting with dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5 not found: ID does not exist" containerID="dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5"
Oct 30 21:18:17.058462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.058391    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5"} err="failed to get container status \"dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5\": rpc error: code = NotFound desc = could not find container \"dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5\": container with ID starting with dd4d531bd46252bd6bea7e42f377cc1a93e720a8ff9bd39455f46683ec563af5 not found: ID does not exist"
Oct 30 21:18:17.122578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.122532    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"]
Oct 30 21:18:17.154265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.154225    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-86671d-5d64dbdb75-tzbwm"]
Oct 30 21:18:17.313452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.313371    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:18:17.313976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.313914    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:18:17.314633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.314617    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:18:17.792055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:17.792024    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cbfce7-e9be-44b8-bed5-9cfd10ac992d" path="/var/lib/kubelet/pods/65cbfce7-e9be-44b8-bed5-9cfd10ac992d/volumes"
Oct 30 21:18:19.807259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.807228    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:18:19.842012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.841978    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-token\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-token\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842075    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-credential-socket\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842125    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-data\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842186    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-certs\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842210    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-envoy\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842248    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istiod-ca-cert\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842295    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-socket\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842364    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-kube-api-access-dtdfs\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-kube-api-access-dtdfs\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842400    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-podinfo\") pod \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\" (UID: \"c4739bd0-7a3e-40e8-bc58-7665486fbce3\") "
Oct 30 21:18:19.842513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842427    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-credential-socket" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:19.842513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842483    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-data" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:19.842782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842607    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-certs" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:19.842782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842715    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-socket" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:19.842893 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.842835    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istiod-ca-cert" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:19.843564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.843448    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.843564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.843474    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.843564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.843500    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.843564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.843524    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.843564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.843539    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.845027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.845001    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-envoy" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:19.845469 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.845449    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-kube-api-access-dtdfs" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "kube-api-access-dtdfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:19.845622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.845465    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-podinfo" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:18:19.845622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.845514    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-token" pod "c4739bd0-7a3e-40e8-bc58-7665486fbce3" (UID: "c4739bd0-7a3e-40e8-bc58-7665486fbce3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:19.944487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.944450    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtdfs\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-kube-api-access-dtdfs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.944487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.944481    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.944487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.944491    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.944756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:19.944500    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4739bd0-7a3e-40e8-bc58-7665486fbce3-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:20.074970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.074880    4633 generic.go:356] "Generic (PLEG): container finished" podID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerID="c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c" exitCode=0
Oct 30 21:18:20.074970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.074925    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" event={"ID":"c4739bd0-7a3e-40e8-bc58-7665486fbce3","Type":"ContainerDied","Data":"c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c"}
Oct 30 21:18:20.074970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.074951    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"
Oct 30 21:18:20.074970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.074970    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc" event={"ID":"c4739bd0-7a3e-40e8-bc58-7665486fbce3","Type":"ContainerDied","Data":"23f601f4db75ce1dbb6200d879e6621d889278ba97c87db22d12127fe33bb0fb"}
Oct 30 21:18:20.075264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.074996    4633 scope.go:122] "RemoveContainer" containerID="c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c"
Oct 30 21:18:20.089628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.089602    4633 scope.go:122] "RemoveContainer" containerID="156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba"
Oct 30 21:18:20.100058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.100034    4633 scope.go:122] "RemoveContainer" containerID="c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c"
Oct 30 21:18:20.100359 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:18:20.100340    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c\": container with ID starting with c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c not found: ID does not exist" containerID="c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c"
Oct 30 21:18:20.100420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.100369    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c"} err="failed to get container status \"c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c\": rpc error: code = NotFound desc = could not find container \"c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c\": container with ID starting with c17fce2603150c4d5a7f34b70e07ccb2043ef13aba1d9a58afa79a8ad8daa36c not found: ID does not exist"
Oct 30 21:18:20.100420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.100388    4633 scope.go:122] "RemoveContainer" containerID="156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba"
Oct 30 21:18:20.100753 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:18:20.100728    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba\": container with ID starting with 156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba not found: ID does not exist" containerID="156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba"
Oct 30 21:18:20.100827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.100760    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba"} err="failed to get container status \"156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba\": rpc error: code = NotFound desc = could not find container \"156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba\": container with ID starting with 156e31e77507f0fc5eec5ac1302245ff999a817449360fbd1f8f44f4fcaee9ba not found: ID does not exist"
Oct 30 21:18:20.103056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.103033    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"]
Oct 30 21:18:20.107340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:20.107318    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6a24e1-openshift-default-6496d74847-c48kc"]
Oct 30 21:18:21.793641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:21.793607    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" path="/var/lib/kubelet/pods/c4739bd0-7a3e-40e8-bc58-7665486fbce3/volumes"
Oct 30 21:18:25.016329 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:18:25.016291    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:18:27.597572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.597532    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7657-6a3a43"]
Oct 30 21:18:27.598418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598395    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy"
Oct 30 21:18:27.598538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598421    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy"
Oct 30 21:18:27.598538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598445    4633 cpu_manager.go:414] "containerMap: removing container" podUID="5d1533f3-2ef5-4499-a0df-5bf52daf6610" containerName="payload-processing"
Oct 30 21:18:27.598538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598452    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1533f3-2ef5-4499-a0df-5bf52daf6610" containerName="payload-processing"
Oct 30 21:18:27.598538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598469    4633 cpu_manager.go:414] "containerMap: removing container" podUID="7f478165-d481-4adc-9113-4c3cbcbb69b6" containerName="payload-pre-processing"
Oct 30 21:18:27.598538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598476    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f478165-d481-4adc-9113-4c3cbcbb69b6" containerName="payload-pre-processing"
Oct 30 21:18:27.598538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598489    4633 cpu_manager.go:414] "containerMap: removing container" podUID="65cbfce7-e9be-44b8-bed5-9cfd10ac992d" containerName="payload-pre-processing"
Oct 30 21:18:27.598538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598495    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cbfce7-e9be-44b8-bed5-9cfd10ac992d" containerName="payload-pre-processing"
Oct 30 21:18:27.598929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598709    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy"
Oct 30 21:18:27.598929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598727    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1533f3-2ef5-4499-a0df-5bf52daf6610" containerName="payload-processing"
Oct 30 21:18:27.598929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598737    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy"
Oct 30 21:18:27.598929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598747    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f478165-d481-4adc-9113-4c3cbcbb69b6" containerName="payload-pre-processing"
Oct 30 21:18:27.598929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.598756    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cbfce7-e9be-44b8-bed5-9cfd10ac992d" containerName="payload-pre-processing"
Oct 30 21:18:27.604333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.604305    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-6a3a43"
Oct 30 21:18:27.607079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.607051    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-6a3a43"]
Oct 30 21:18:27.607860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.607821    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9lmpp\""
Oct 30 21:18:27.684355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.684317    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftsnq\" (UniqueName: \"kubernetes.io/projected/977d4d58-7d29-4a04-ac3b-b91533d16991-kube-api-access-ftsnq\") pod \"test-curl-7657-6a3a43\" (UID: \"977d4d58-7d29-4a04-ac3b-b91533d16991\") " pod="opendatahub/test-curl-7657-6a3a43"
Oct 30 21:18:27.824126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.824083    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftsnq\" (UniqueName: \"kubernetes.io/projected/977d4d58-7d29-4a04-ac3b-b91533d16991-kube-api-access-ftsnq\") pod \"test-curl-7657-6a3a43\" (UID: \"977d4d58-7d29-4a04-ac3b-b91533d16991\") " pod="opendatahub/test-curl-7657-6a3a43"
Oct 30 21:18:27.841523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.841483    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftsnq\" (UniqueName: \"kubernetes.io/projected/977d4d58-7d29-4a04-ac3b-b91533d16991-kube-api-access-ftsnq\") pod \"test-curl-7657-6a3a43\" (UID: \"977d4d58-7d29-4a04-ac3b-b91533d16991\") " pod="opendatahub/test-curl-7657-6a3a43"
Oct 30 21:18:27.919834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:27.919800    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-6a3a43"
Oct 30 21:18:28.203086 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:18:28.203054    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977d4d58_7d29_4a04_ac3b_b91533d16991.slice/crio-91485f622e6f2abc5f8dcc053bf37bb0e7f5beead6b3fcdc08b50c60af547882 WatchSource:0}: Error finding container 91485f622e6f2abc5f8dcc053bf37bb0e7f5beead6b3fcdc08b50c60af547882: Status 404 returned error can't find the container with id 91485f622e6f2abc5f8dcc053bf37bb0e7f5beead6b3fcdc08b50c60af547882
Oct 30 21:18:28.203639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.203617    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-6a3a43"]
Oct 30 21:18:28.341965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.341865    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"]
Oct 30 21:18:28.342594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.342556    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4" podUID="d242aa53-5566-4147-a390-1200dd699848" containerName="maas-api" containerID="cri-o://92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170" gracePeriod=30
Oct 30 21:18:28.801714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.797100    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:18:28.868161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.868130    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d242aa53-5566-4147-a390-1200dd699848-kube-api-access-hwvr7\" (UniqueName: \"kubernetes.io/projected/d242aa53-5566-4147-a390-1200dd699848-kube-api-access-hwvr7\") pod \"d242aa53-5566-4147-a390-1200dd699848\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") "
Oct 30 21:18:28.868345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.868201    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-maas-api-tls\") pod \"d242aa53-5566-4147-a390-1200dd699848\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") "
Oct 30 21:18:28.868345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.868247    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d242aa53-5566-4147-a390-1200dd699848-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d242aa53-5566-4147-a390-1200dd699848-openshift-service-ca\") pod \"d242aa53-5566-4147-a390-1200dd699848\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") "
Oct 30 21:18:28.868345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.868319    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-metrics-certs\") pod \"d242aa53-5566-4147-a390-1200dd699848\" (UID: \"d242aa53-5566-4147-a390-1200dd699848\") "
Oct 30 21:18:28.868974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.868734    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d242aa53-5566-4147-a390-1200dd699848-openshift-service-ca" pod "d242aa53-5566-4147-a390-1200dd699848" (UID: "d242aa53-5566-4147-a390-1200dd699848"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:28.868974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.868868    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d242aa53-5566-4147-a390-1200dd699848-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:28.871192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.871127    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d242aa53-5566-4147-a390-1200dd699848-kube-api-access-hwvr7" pod "d242aa53-5566-4147-a390-1200dd699848" (UID: "d242aa53-5566-4147-a390-1200dd699848"). InnerVolumeSpecName "kube-api-access-hwvr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:28.871192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.871128    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-metrics-certs" pod "d242aa53-5566-4147-a390-1200dd699848" (UID: "d242aa53-5566-4147-a390-1200dd699848"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:28.871367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.871349    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-maas-api-tls" pod "d242aa53-5566-4147-a390-1200dd699848" (UID: "d242aa53-5566-4147-a390-1200dd699848"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:28.976713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.970943    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvr7\" (UniqueName: \"kubernetes.io/projected/d242aa53-5566-4147-a390-1200dd699848-kube-api-access-hwvr7\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:28.976713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.970990    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:28.976713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:28.971007    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d242aa53-5566-4147-a390-1200dd699848-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:29.162433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.162399    4633 generic.go:356] "Generic (PLEG): container finished" podID="d242aa53-5566-4147-a390-1200dd699848" containerID="92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170" exitCode=0
Oct 30 21:18:29.162634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.162462    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"
Oct 30 21:18:29.162634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.162487    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4" event={"ID":"d242aa53-5566-4147-a390-1200dd699848","Type":"ContainerDied","Data":"92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170"}
Oct 30 21:18:29.162634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.162581    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4" event={"ID":"d242aa53-5566-4147-a390-1200dd699848","Type":"ContainerDied","Data":"d977209a31ab1ca9b3ae4555d21794db3ea4cc1c365b1c8e4151310f637e1c14"}
Oct 30 21:18:29.162634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.162610    4633 scope.go:122] "RemoveContainer" containerID="92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170"
Oct 30 21:18:29.164097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.164076    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-6a3a43" event={"ID":"977d4d58-7d29-4a04-ac3b-b91533d16991","Type":"ContainerStarted","Data":"91485f622e6f2abc5f8dcc053bf37bb0e7f5beead6b3fcdc08b50c60af547882"}
Oct 30 21:18:29.174891 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.174873    4633 scope.go:122] "RemoveContainer" containerID="92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170"
Oct 30 21:18:29.175194 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:18:29.175176    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170\": container with ID starting with 92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170 not found: ID does not exist" containerID="92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170"
Oct 30 21:18:29.175258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.175205    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170"} err="failed to get container status \"92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170\": rpc error: code = NotFound desc = could not find container \"92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170\": container with ID starting with 92aae0e199b7777710b8b8dd22fc0762bf29c057c3c9722c10588857daa63170 not found: ID does not exist"
Oct 30 21:18:29.187566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.187524    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"]
Oct 30 21:18:29.191888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.191862    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86671d-bbf98c7bc-wcdj4"]
Oct 30 21:18:29.794513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:29.794461    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d242aa53-5566-4147-a390-1200dd699848" path="/var/lib/kubelet/pods/d242aa53-5566-4147-a390-1200dd699848/volumes"
Oct 30 21:18:30.061455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.061379    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vbsd2-tn5zx"]
Oct 30 21:18:30.063065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.063039    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy"
Oct 30 21:18:30.063248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.063223    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4739bd0-7a3e-40e8-bc58-7665486fbce3" containerName="istio-proxy"
Oct 30 21:18:30.063427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.063311    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d242aa53-5566-4147-a390-1200dd699848" containerName="maas-api"
Oct 30 21:18:30.063427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.063322    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d242aa53-5566-4147-a390-1200dd699848" containerName="maas-api"
Oct 30 21:18:30.063638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.063525    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d242aa53-5566-4147-a390-1200dd699848" containerName="maas-api"
Oct 30 21:18:30.091638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.091608    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vbsd2-tn5zx"]
Oct 30 21:18:30.091850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.091738    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.099557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.099529    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:18:30.099762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.099618    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dcwvc\""
Oct 30 21:18:30.104375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.104351    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7tdl\" (UniqueName: \"kubernetes.io/projected/95881bbb-04ab-44f1-a15b-f0c90ca72d84-kube-api-access-k7tdl\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.104484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.104402    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95881bbb-04ab-44f1-a15b-f0c90ca72d84-archives-path\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.104575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.104545    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-service-ca-bundle\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.104632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.104585    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-trusted-ca-bundle\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.105040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.105000    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:18:30.205899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.205866    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-service-ca-bundle\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.206093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.205908    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-trusted-ca-bundle\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.206093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.205997    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7tdl\" (UniqueName: \"kubernetes.io/projected/95881bbb-04ab-44f1-a15b-f0c90ca72d84-kube-api-access-k7tdl\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.206093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.206046    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95881bbb-04ab-44f1-a15b-f0c90ca72d84-archives-path\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.206528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.206499    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95881bbb-04ab-44f1-a15b-f0c90ca72d84-archives-path\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.206658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.206613    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-service-ca-bundle\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.206778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.206760    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-trusted-ca-bundle\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.215900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.215870    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7tdl\" (UniqueName: \"kubernetes.io/projected/95881bbb-04ab-44f1-a15b-f0c90ca72d84-kube-api-access-k7tdl\") pod \"periodic-gathering-vbsd2-tn5zx\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") " pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.410726 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.410689    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:18:30.771294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:30.771263    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vbsd2-tn5zx"]
Oct 30 21:18:30.773086 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:18:30.773061    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95881bbb_04ab_44f1_a15b_f0c90ca72d84.slice/crio-9d1ca9e61e00c4e0b6df86fdbd9a31b14c1d30649a592688629af5f45f6d70f6 WatchSource:0}: Error finding container 9d1ca9e61e00c4e0b6df86fdbd9a31b14c1d30649a592688629af5f45f6d70f6: Status 404 returned error can't find the container with id 9d1ca9e61e00c4e0b6df86fdbd9a31b14c1d30649a592688629af5f45f6d70f6
Oct 30 21:18:31.191492 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:31.191449    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-6a3a43" event={"ID":"977d4d58-7d29-4a04-ac3b-b91533d16991","Type":"ContainerStarted","Data":"79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60"}
Oct 30 21:18:31.194255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:31.194213    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx" event={"ID":"95881bbb-04ab-44f1-a15b-f0c90ca72d84","Type":"ContainerStarted","Data":"06bcaacbf84a25eb5adc3e97e017e47591d97c0d11e6168f589b2801857e5cdd"}
Oct 30 21:18:31.194397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:31.194281    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx" event={"ID":"95881bbb-04ab-44f1-a15b-f0c90ca72d84","Type":"ContainerStarted","Data":"9d1ca9e61e00c4e0b6df86fdbd9a31b14c1d30649a592688629af5f45f6d70f6"}
Oct 30 21:18:31.208994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:31.208933    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7657-6a3a43" podStartSLOduration=1.556955517 podStartE2EDuration="4.208914239s" totalImagesPullingTime="2.651958722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:31.204934397 +0000 UTC m=+1019.554661796" watchObservedRunningTime="2026-10-30 21:18:31.208914239 +0000 UTC m=+1019.558641637"
Oct 30 21:18:31.223395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:31.223338    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx" podStartSLOduration=1.223320216 podStartE2EDuration="1.223320216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:31.222790291 +0000 UTC m=+1019.572517687" watchObservedRunningTime="2026-10-30 21:18:31.223320216 +0000 UTC m=+1019.573047618"
Oct 30 21:18:31.883162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:31.882986    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7657-6a3a43"]
Oct 30 21:18:31.895476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:31.895443    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7657-6a3a43"]
Oct 30 21:18:33.210860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:33.210822    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7657-6a3a43" podUID="977d4d58-7d29-4a04-ac3b-b91533d16991" containerName="test-curl-7657-6a3a43" containerID="cri-o://79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60" gracePeriod=2
Oct 30 21:18:33.213179 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:18:33.213152    4633 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7657-6a3a43\" is forbidden: User \"system:node:ip-10-0-140-12.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-12.us-east-2.compute.internal' and this object" podUID="977d4d58-7d29-4a04-ac3b-b91533d16991" pod="opendatahub/test-curl-7657-6a3a43"
Oct 30 21:18:35.034201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.034166    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"]
Oct 30 21:18:35.034764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.034440    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8" podUID="f6e9614a-bfed-4d4d-97f3-41b5713da79d" containerName="maas-api" containerID="cri-o://dfdd96291c76e5aa36b2a43278ee7f4f938b3ded431f182cb84a7624071fe9ec" gracePeriod=30
Oct 30 21:18:35.237688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.236035    4633 generic.go:356] "Generic (PLEG): container finished" podID="f6e9614a-bfed-4d4d-97f3-41b5713da79d" containerID="dfdd96291c76e5aa36b2a43278ee7f4f938b3ded431f182cb84a7624071fe9ec" exitCode=0
Oct 30 21:18:35.237688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.236090    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8" event={"ID":"f6e9614a-bfed-4d4d-97f3-41b5713da79d","Type":"ContainerDied","Data":"dfdd96291c76e5aa36b2a43278ee7f4f938b3ded431f182cb84a7624071fe9ec"}
Oct 30 21:18:35.468274 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.468135    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:18:35.495890 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.495809    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e9614a-bfed-4d4d-97f3-41b5713da79d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9614a-bfed-4d4d-97f3-41b5713da79d-openshift-service-ca\") pod \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") "
Oct 30 21:18:35.496063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.495950    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-metrics-certs\") pod \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") "
Oct 30 21:18:35.496063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.496016    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e9614a-bfed-4d4d-97f3-41b5713da79d-kube-api-access-ldbzr\" (UniqueName: \"kubernetes.io/projected/f6e9614a-bfed-4d4d-97f3-41b5713da79d-kube-api-access-ldbzr\") pod \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") "
Oct 30 21:18:35.496194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.496077    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-maas-api-tls\") pod \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\" (UID: \"f6e9614a-bfed-4d4d-97f3-41b5713da79d\") "
Oct 30 21:18:35.551741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.541222    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e9614a-bfed-4d4d-97f3-41b5713da79d-openshift-service-ca" pod "f6e9614a-bfed-4d4d-97f3-41b5713da79d" (UID: "f6e9614a-bfed-4d4d-97f3-41b5713da79d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:35.579731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.579584    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-metrics-certs" pod "f6e9614a-bfed-4d4d-97f3-41b5713da79d" (UID: "f6e9614a-bfed-4d4d-97f3-41b5713da79d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:35.580076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.580037    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-maas-api-tls" pod "f6e9614a-bfed-4d4d-97f3-41b5713da79d" (UID: "f6e9614a-bfed-4d4d-97f3-41b5713da79d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:35.582311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.582270    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e9614a-bfed-4d4d-97f3-41b5713da79d-kube-api-access-ldbzr" pod "f6e9614a-bfed-4d4d-97f3-41b5713da79d" (UID: "f6e9614a-bfed-4d4d-97f3-41b5713da79d"). InnerVolumeSpecName "kube-api-access-ldbzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:35.606488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.602425    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e9614a-bfed-4d4d-97f3-41b5713da79d-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.606488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.602532    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.606488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.602553    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldbzr\" (UniqueName: \"kubernetes.io/projected/f6e9614a-bfed-4d4d-97f3-41b5713da79d-kube-api-access-ldbzr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.606488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.602569    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6e9614a-bfed-4d4d-97f3-41b5713da79d-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.656201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.656172    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-6a3a43"
Oct 30 21:18:35.703133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.703102    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977d4d58-7d29-4a04-ac3b-b91533d16991-kube-api-access-ftsnq\" (UniqueName: \"kubernetes.io/projected/977d4d58-7d29-4a04-ac3b-b91533d16991-kube-api-access-ftsnq\") pod \"977d4d58-7d29-4a04-ac3b-b91533d16991\" (UID: \"977d4d58-7d29-4a04-ac3b-b91533d16991\") "
Oct 30 21:18:35.705801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.705767    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977d4d58-7d29-4a04-ac3b-b91533d16991-kube-api-access-ftsnq" pod "977d4d58-7d29-4a04-ac3b-b91533d16991" (UID: "977d4d58-7d29-4a04-ac3b-b91533d16991"). InnerVolumeSpecName "kube-api-access-ftsnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:35.839755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.813885    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftsnq\" (UniqueName: \"kubernetes.io/projected/977d4d58-7d29-4a04-ac3b-b91533d16991-kube-api-access-ftsnq\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:35.940309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:35.940274    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977d4d58-7d29-4a04-ac3b-b91533d16991" path="/var/lib/kubelet/pods/977d4d58-7d29-4a04-ac3b-b91533d16991/volumes"
Oct 30 21:18:36.244258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.244226    4633 generic.go:356] "Generic (PLEG): container finished" podID="977d4d58-7d29-4a04-ac3b-b91533d16991" containerID="79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60" exitCode=137
Oct 30 21:18:36.244747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.244274    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-6a3a43"
Oct 30 21:18:36.244747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.244319    4633 scope.go:122] "RemoveContainer" containerID="79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60"
Oct 30 21:18:36.245938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.245921    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"
Oct 30 21:18:36.246042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.245965    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8" event={"ID":"f6e9614a-bfed-4d4d-97f3-41b5713da79d","Type":"ContainerDied","Data":"a7e2c81007eec8faa254655631df45076e3be5c2f7db6136ea630e6be7b0f318"}
Oct 30 21:18:36.258470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.258451    4633 scope.go:122] "RemoveContainer" containerID="79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60"
Oct 30 21:18:36.258814 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:18:36.258794    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60\": container with ID starting with 79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60 not found: ID does not exist" containerID="79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60"
Oct 30 21:18:36.258880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.258822    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60"} err="failed to get container status \"79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60\": rpc error: code = NotFound desc = could not find container \"79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60\": container with ID starting with 79467cdf5b2fa0a097c82f4dc74c80f500beeecfe9cd1d27627b31d82e01aa60 not found: ID does not exist"
Oct 30 21:18:36.258880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.258838    4633 scope.go:122] "RemoveContainer" containerID="dfdd96291c76e5aa36b2a43278ee7f4f938b3ded431f182cb84a7624071fe9ec"
Oct 30 21:18:36.269467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.269445    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"]
Oct 30 21:18:36.274292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:36.274268    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c499bf-986fbcc77-2kkm8"]
Oct 30 21:18:37.115659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.115620    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"]
Oct 30 21:18:37.116372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.116351    4633 cpu_manager.go:414] "containerMap: removing container" podUID="977d4d58-7d29-4a04-ac3b-b91533d16991" containerName="test-curl-7657-6a3a43"
Oct 30 21:18:37.116476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.116373    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977d4d58-7d29-4a04-ac3b-b91533d16991" containerName="test-curl-7657-6a3a43"
Oct 30 21:18:37.116476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.116395    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f6e9614a-bfed-4d4d-97f3-41b5713da79d" containerName="maas-api"
Oct 30 21:18:37.116476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.116402    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e9614a-bfed-4d4d-97f3-41b5713da79d" containerName="maas-api"
Oct 30 21:18:37.116621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.116590    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e9614a-bfed-4d4d-97f3-41b5713da79d" containerName="maas-api"
Oct 30 21:18:37.116621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.116617    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="977d4d58-7d29-4a04-ac3b-b91533d16991" containerName="test-curl-7657-6a3a43"
Oct 30 21:18:37.139656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.139617    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.145937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.145898    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"]
Oct 30 21:18:37.146798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.146775    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0c4aac-openshift-default-dockercfg-jlzfz\""
Oct 30 21:18:37.258793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.258748    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-certs\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.268355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.258872    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-credential-socket\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.268355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.258919    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-socket\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.268355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.259070    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istiod-ca-cert\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.268355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.259199    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628wp\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-kube-api-access-628wp\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.268355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.259325    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-podinfo\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.268355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.259384    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-token\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.268355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.259456    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-envoy\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.268355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.259479    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-data\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.364813    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-certs\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.364895    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-credential-socket\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.364935    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-socket\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.364984    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istiod-ca-cert\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.365075    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628wp\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-kube-api-access-628wp\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.365199    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-podinfo\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.365258    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-token\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.365333    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-envoy\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.365746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.365356    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-data\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.366387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.365906    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-data\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.366387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.366148    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-certs\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.366387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.366382    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-credential-socket\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.370703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.366615    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-socket\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.370703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.367335    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istiod-ca-cert\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.394166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.394125    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-envoy\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.397021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.395178    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-podinfo\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.402066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.401978    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-token\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.436424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.436388    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-628wp\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-kube-api-access-628wp\") pod \"e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") " pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.498138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.497564    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:37.591861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.591815    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:18:37.793727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.793692    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e9614a-bfed-4d4d-97f3-41b5713da79d" path="/var/lib/kubelet/pods/f6e9614a-bfed-4d4d-97f3-41b5713da79d/volumes"
Oct 30 21:18:37.858207 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:18:37.858178    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f008a7d_ba68_4238_9dee_3bc3d6b56a9f.slice/crio-0ec42d30da01fa8e38f1d8d97ee4ee156fdb3d84dc42c94c7fb053cb95406a35 WatchSource:0}: Error finding container 0ec42d30da01fa8e38f1d8d97ee4ee156fdb3d84dc42c94c7fb053cb95406a35: Status 404 returned error can't find the container with id 0ec42d30da01fa8e38f1d8d97ee4ee156fdb3d84dc42c94c7fb053cb95406a35
Oct 30 21:18:37.861730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.861686    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:37.861856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.861788    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:37.861856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.861835    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:18:37.862454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:37.862349    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"]
Oct 30 21:18:38.290757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:38.290717    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" event={"ID":"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f","Type":"ContainerStarted","Data":"405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff"}
Oct 30 21:18:38.291170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:38.290785    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" event={"ID":"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f","Type":"ContainerStarted","Data":"0ec42d30da01fa8e38f1d8d97ee4ee156fdb3d84dc42c94c7fb053cb95406a35"}
Oct 30 21:18:38.337537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:38.337474    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" podStartSLOduration=1.337454298 podStartE2EDuration="1.337454298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:38.331736836 +0000 UTC m=+1026.681464234" watchObservedRunningTime="2026-10-30 21:18:38.337454298 +0000 UTC m=+1026.687181696"
Oct 30 21:18:38.514693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:38.502514    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:38.629751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:38.629473    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:39.333938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:39.333901    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:39.336743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:39.336653    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:18:42.426650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.426572    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"]
Oct 30 21:18:42.431360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.431330    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:42.446714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.446663    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"]
Oct 30 21:18:42.514723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.504149    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0c4aac\""
Oct 30 21:18:42.514723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.504314    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0c4aac-dockercfg-bvx8n\""
Oct 30 21:18:42.566967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.566926    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vk7r\" (UniqueName: \"kubernetes.io/projected/f590b8fa-db79-49f5-b40d-5386e2694fa9-kube-api-access-8vk7r\") pod \"payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd\" (UID: \"f590b8fa-db79-49f5-b40d-5386e2694fa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:42.593229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.593177    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f590b8fa-db79-49f5-b40d-5386e2694fa9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd\" (UID: \"f590b8fa-db79-49f5-b40d-5386e2694fa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:42.705741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.703932    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vk7r\" (UniqueName: \"kubernetes.io/projected/f590b8fa-db79-49f5-b40d-5386e2694fa9-kube-api-access-8vk7r\") pod \"payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd\" (UID: \"f590b8fa-db79-49f5-b40d-5386e2694fa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:42.705741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.704113    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f590b8fa-db79-49f5-b40d-5386e2694fa9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd\" (UID: \"f590b8fa-db79-49f5-b40d-5386e2694fa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:42.705741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.705166    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f590b8fa-db79-49f5-b40d-5386e2694fa9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd\" (UID: \"f590b8fa-db79-49f5-b40d-5386e2694fa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:42.728788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.728750    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vk7r\" (UniqueName: \"kubernetes.io/projected/f590b8fa-db79-49f5-b40d-5386e2694fa9-kube-api-access-8vk7r\") pod \"payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd\" (UID: \"f590b8fa-db79-49f5-b40d-5386e2694fa9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:42.752572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:42.752540    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:43.118698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:43.114574    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"]
Oct 30 21:18:43.184405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:43.184370    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"]
Oct 30 21:18:43.185045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:43.184989    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" containerID="cri-o://6614994fd3b3f3c94d301472471a3c426bcf7c031f02209cdb91bbac98b7cf14" gracePeriod=30
Oct 30 21:18:43.425065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:43.425031    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd" event={"ID":"f590b8fa-db79-49f5-b40d-5386e2694fa9","Type":"ContainerStarted","Data":"f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df"}
Oct 30 21:18:43.425242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:43.425089    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd" event={"ID":"f590b8fa-db79-49f5-b40d-5386e2694fa9","Type":"ContainerStarted","Data":"bf9e59e0226e6e79296d86b82824c6bf8d7c21612a591626e19f6f665bdffc74"}
Oct 30 21:18:43.425242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:43.425137    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:43.448279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:43.448234    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd" podStartSLOduration=1.4482225930000001 podStartE2EDuration="1.448222593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:43.446527832 +0000 UTC m=+1031.796255223" watchObservedRunningTime="2026-10-30 21:18:43.448222593 +0000 UTC m=+1031.797950019"
Oct 30 21:18:47.326696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.320160    4633 patch_prober.go:29] interesting pod/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:18:47.326696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.320222    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:18:47.459335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.459302    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"]
Oct 30 21:18:47.464043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.464016    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.469799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.469775    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0c4aac-serving-cert\""
Oct 30 21:18:47.480250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.480221    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"]
Oct 30 21:18:47.611762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.608445    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqf2\" (UniqueName: \"kubernetes.io/projected/ce3f7242-71a0-410f-8923-c4fdea2c393c-kube-api-access-pxqf2\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.611762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.608589    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce3f7242-71a0-410f-8923-c4fdea2c393c-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.611762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.608634    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-metrics-certs\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.611762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.608843    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-maas-api-tls\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.710311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.710273    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-maas-api-tls\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.710496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.710368    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqf2\" (UniqueName: \"kubernetes.io/projected/ce3f7242-71a0-410f-8923-c4fdea2c393c-kube-api-access-pxqf2\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.710496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.710440    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce3f7242-71a0-410f-8923-c4fdea2c393c-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.710496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.710480    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-metrics-certs\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.711365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.711329    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce3f7242-71a0-410f-8923-c4fdea2c393c-openshift-service-ca\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.713233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.713207    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-maas-api-tls\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.714213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.714186    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-metrics-certs\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.718353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.718331    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqf2\" (UniqueName: \"kubernetes.io/projected/ce3f7242-71a0-410f-8923-c4fdea2c393c-kube-api-access-pxqf2\") pod \"maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:47.795325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:47.795284    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:48.440816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.440788    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"]
Oct 30 21:18:48.507578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.507487    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs" event={"ID":"ce3f7242-71a0-410f-8923-c4fdea2c393c","Type":"ContainerStarted","Data":"d133908e29be0593dfaa099514198f8b9427ea4c621c36172b22b8b244272dd7"}
Oct 30 21:18:48.510161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.510131    4633 generic.go:356] "Generic (PLEG): container finished" podID="55172deb-27cf-4d37-8919-dc78904d47f7" containerID="6614994fd3b3f3c94d301472471a3c426bcf7c031f02209cdb91bbac98b7cf14" exitCode=0
Oct 30 21:18:48.510282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.510173    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" event={"ID":"55172deb-27cf-4d37-8919-dc78904d47f7","Type":"ContainerDied","Data":"6614994fd3b3f3c94d301472471a3c426bcf7c031f02209cdb91bbac98b7cf14"}
Oct 30 21:18:48.510282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.510231    4633 scope.go:122] "RemoveContainer" containerID="8b16d78c2e74e24095c682f676aabe0f2d4631cb517f26915bad4e2423413c77"
Oct 30 21:18:48.523214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.523185    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:18:48.662634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.662599    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-istio-token\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-istio-token\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.662881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.662657    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-envoy\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.663229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.663202    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55172deb-27cf-4d37-8919-dc78904d47f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55172deb-27cf-4d37-8919-dc78904d47f7-istio-podinfo\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.663352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.663329    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-kube-api-access-b45g6\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-kube-api-access-b45g6\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.663435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.663419    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-certs\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.663538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.663524    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-data\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.663616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.663577    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-credential-socket\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.663847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.663638    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-socket\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.663847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.663688    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55172deb-27cf-4d37-8919-dc78904d47f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55172deb-27cf-4d37-8919-dc78904d47f7-istiod-ca-cert\") pod \"55172deb-27cf-4d37-8919-dc78904d47f7\" (UID: \"55172deb-27cf-4d37-8919-dc78904d47f7\") "
Oct 30 21:18:48.664583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.664559    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55172deb-27cf-4d37-8919-dc78904d47f7-istiod-ca-cert" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:48.665233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.665210    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-certs" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:48.668178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.668156    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-istio-token" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:48.668505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.668485    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-data" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:48.668789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.668768    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-credential-socket" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:48.669079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.669057    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-socket" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:48.671728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.671705    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-envoy" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:48.674500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.674463    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55172deb-27cf-4d37-8919-dc78904d47f7-istio-podinfo" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:18:48.677299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.677264    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-kube-api-access-b45g6" pod "55172deb-27cf-4d37-8919-dc78904d47f7" (UID: "55172deb-27cf-4d37-8919-dc78904d47f7"). InnerVolumeSpecName "kube-api-access-b45g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:48.765719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765642    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:48.765719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765708    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:48.765719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765719    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:48.765927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765729    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:48.765927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765738    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55172deb-27cf-4d37-8919-dc78904d47f7-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:48.765927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765747    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:48.765927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765755    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55172deb-27cf-4d37-8919-dc78904d47f7-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:48.765927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765763    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55172deb-27cf-4d37-8919-dc78904d47f7-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:48.765927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:48.765771    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b45g6\" (UniqueName: \"kubernetes.io/projected/55172deb-27cf-4d37-8919-dc78904d47f7-kube-api-access-b45g6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:49.520118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.520082    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs" event={"ID":"ce3f7242-71a0-410f-8923-c4fdea2c393c","Type":"ContainerStarted","Data":"7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904"}
Oct 30 21:18:49.520547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.520141    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:18:49.521632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.521608    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct" event={"ID":"55172deb-27cf-4d37-8919-dc78904d47f7","Type":"ContainerDied","Data":"b5015159b027fe3e6cdcc31994abf2492e06c18cb446944ee2ac8e5bf6cf9ddf"}
Oct 30 21:18:49.521632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.521623    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"
Oct 30 21:18:49.521819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.521661    4633 scope.go:122] "RemoveContainer" containerID="6614994fd3b3f3c94d301472471a3c426bcf7c031f02209cdb91bbac98b7cf14"
Oct 30 21:18:49.541759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.541705    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs" podStartSLOduration=2.124473975 podStartE2EDuration="2.541690734s" totalImagesPullingTime="417.216759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:49.537613963 +0000 UTC m=+1037.887341340" watchObservedRunningTime="2026-10-30 21:18:49.541690734 +0000 UTC m=+1037.891418127"
Oct 30 21:18:49.580455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.580417    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"]
Oct 30 21:18:49.584479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.584443    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c499bf-openshift-default-7b7f4dfc49-w4hct"]
Oct 30 21:18:49.792650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:49.792571    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" path="/var/lib/kubelet/pods/55172deb-27cf-4d37-8919-dc78904d47f7/volumes"
Oct 30 21:18:54.437062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:54.437030    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:18:55.015703 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:18:55.015630    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:18:55.536870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:18:55.536840    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:19:02.000505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:01.999166    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:19:03.032853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:03.032804    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"]
Oct 30 21:19:03.048100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:03.048033    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr" podUID="6844674a-d6eb-4b58-94db-c77eec954d6f" containerName="maas-api" containerID="cri-o://8e76caa43c7ae0ec4db876575371ef903eb02e024431b92879e9c29bbcf601bd" gracePeriod=30
Oct 30 21:19:03.790541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:03.790502    4633 generic.go:356] "Generic (PLEG): container finished" podID="6844674a-d6eb-4b58-94db-c77eec954d6f" containerID="8e76caa43c7ae0ec4db876575371ef903eb02e024431b92879e9c29bbcf601bd" exitCode=0
Oct 30 21:19:03.791052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:03.791020    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr" event={"ID":"6844674a-d6eb-4b58-94db-c77eec954d6f","Type":"ContainerDied","Data":"8e76caa43c7ae0ec4db876575371ef903eb02e024431b92879e9c29bbcf601bd"}
Oct 30 21:19:04.585318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:04.584535    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:04.749777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:04.743938    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-msql6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:19:04.749777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:04.744007    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-msql6" podUID="bfa29907-7f35-4ba2-8303-54079a98cb5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:05.060812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.055221    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr" event={"ID":"6844674a-d6eb-4b58-94db-c77eec954d6f","Type":"ContainerDied","Data":"80f377d97726297d1ca92ad5021e5efb0758d3d42dc9ec532c831cbd7a1c1db3"}
Oct 30 21:19:05.060812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.055286    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80f377d97726297d1ca92ad5021e5efb0758d3d42dc9ec532c831cbd7a1c1db3"
Oct 30 21:19:05.188696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.188647    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:19:05.265701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.241043    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-metrics-certs\") pod \"6844674a-d6eb-4b58-94db-c77eec954d6f\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") "
Oct 30 21:19:05.384096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.383986    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6844674a-d6eb-4b58-94db-c77eec954d6f-kube-api-access-dj8vh\" (UniqueName: \"kubernetes.io/projected/6844674a-d6eb-4b58-94db-c77eec954d6f-kube-api-access-dj8vh\") pod \"6844674a-d6eb-4b58-94db-c77eec954d6f\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") "
Oct 30 21:19:05.398703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.387875    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-maas-api-tls\") pod \"6844674a-d6eb-4b58-94db-c77eec954d6f\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") "
Oct 30 21:19:05.406698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.399018    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6844674a-d6eb-4b58-94db-c77eec954d6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6844674a-d6eb-4b58-94db-c77eec954d6f-openshift-service-ca\") pod \"6844674a-d6eb-4b58-94db-c77eec954d6f\" (UID: \"6844674a-d6eb-4b58-94db-c77eec954d6f\") "
Oct 30 21:19:05.475948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.302211    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-metrics-certs" pod "6844674a-d6eb-4b58-94db-c77eec954d6f" (UID: "6844674a-d6eb-4b58-94db-c77eec954d6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:05.506842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.411117    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6844674a-d6eb-4b58-94db-c77eec954d6f-openshift-service-ca" pod "6844674a-d6eb-4b58-94db-c77eec954d6f" (UID: "6844674a-d6eb-4b58-94db-c77eec954d6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:05.551701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.544420    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:05.551997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.551971    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6844674a-d6eb-4b58-94db-c77eec954d6f-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:05.598304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.591936    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:05.598304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.593590    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:19:05.616857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.616778    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-maas-api-tls" pod "6844674a-d6eb-4b58-94db-c77eec954d6f" (UID: "6844674a-d6eb-4b58-94db-c77eec954d6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:05.638813 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.638729    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6844674a-d6eb-4b58-94db-c77eec954d6f-kube-api-access-dj8vh" pod "6844674a-d6eb-4b58-94db-c77eec954d6f" (UID: "6844674a-d6eb-4b58-94db-c77eec954d6f"). InnerVolumeSpecName "kube-api-access-dj8vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:05.715773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.715540    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj8vh\" (UniqueName: \"kubernetes.io/projected/6844674a-d6eb-4b58-94db-c77eec954d6f-kube-api-access-dj8vh\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:05.715773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:05.715581    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6844674a-d6eb-4b58-94db-c77eec954d6f-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:06.008129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:06.008092    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:19:06.108693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:06.101275    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"
Oct 30 21:19:06.318648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:06.318565    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"]
Oct 30 21:19:06.391629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:06.391549    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr"]
Oct 30 21:19:07.748645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.746906    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3d1de-54d7cc9b75-rwnsr" podUID="6844674a-d6eb-4b58-94db-c77eec954d6f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.113:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:19:07.764244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.764215    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"]
Oct 30 21:19:07.764883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.764869    4633 cpu_manager.go:414] "containerMap: removing container" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy"
Oct 30 21:19:07.764934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.764885    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy"
Oct 30 21:19:07.764934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.764906    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6844674a-d6eb-4b58-94db-c77eec954d6f" containerName="maas-api"
Oct 30 21:19:07.764934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.764912    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6844674a-d6eb-4b58-94db-c77eec954d6f" containerName="maas-api"
Oct 30 21:19:07.765029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.765025    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy"
Oct 30 21:19:07.765058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.765033    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy"
Oct 30 21:19:07.765058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.765042    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6844674a-d6eb-4b58-94db-c77eec954d6f" containerName="maas-api"
Oct 30 21:19:07.770276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.770254    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:07.773816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.773791    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"]
Oct 30 21:19:07.774715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.774695    4633 cpu_manager.go:414] "containerMap: removing container" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy"
Oct 30 21:19:07.774814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.774716    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55172deb-27cf-4d37-8919-dc78904d47f7" containerName="istio-proxy"
Oct 30 21:19:07.778643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.778606    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e494a9\""
Oct 30 21:19:07.778798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.778741    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e494a9-dockercfg-k8vmt\""
Oct 30 21:19:07.779157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.779139    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:07.780407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.780381    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"]
Oct 30 21:19:07.809426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.809400    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6844674a-d6eb-4b58-94db-c77eec954d6f" path="/var/lib/kubelet/pods/6844674a-d6eb-4b58-94db-c77eec954d6f/volumes"
Oct 30 21:19:07.810090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.810069    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"]
Oct 30 21:19:07.874823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.867426    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281b139e-c545-4213-ab8b-b83fafff473e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86\" (UID: \"281b139e-c545-4213-ab8b-b83fafff473e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:07.890708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.881074    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42781959-fe94-4542-aa6c-57f3b7a3e180-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8\" (UID: \"42781959-fe94-4542-aa6c-57f3b7a3e180\") " pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:07.902364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.891171    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm26v\" (UniqueName: \"kubernetes.io/projected/42781959-fe94-4542-aa6c-57f3b7a3e180-kube-api-access-wm26v\") pod \"payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8\" (UID: \"42781959-fe94-4542-aa6c-57f3b7a3e180\") " pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:07.916699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:07.908060    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm55j\" (UniqueName: \"kubernetes.io/projected/281b139e-c545-4213-ab8b-b83fafff473e-kube-api-access-pm55j\") pod \"payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86\" (UID: \"281b139e-c545-4213-ab8b-b83fafff473e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:08.020742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.012824    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42781959-fe94-4542-aa6c-57f3b7a3e180-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8\" (UID: \"42781959-fe94-4542-aa6c-57f3b7a3e180\") " pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:08.020742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.012996    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm26v\" (UniqueName: \"kubernetes.io/projected/42781959-fe94-4542-aa6c-57f3b7a3e180-kube-api-access-wm26v\") pod \"payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8\" (UID: \"42781959-fe94-4542-aa6c-57f3b7a3e180\") " pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:08.020742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.013443    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm55j\" (UniqueName: \"kubernetes.io/projected/281b139e-c545-4213-ab8b-b83fafff473e-kube-api-access-pm55j\") pod \"payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86\" (UID: \"281b139e-c545-4213-ab8b-b83fafff473e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:08.020742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.013539    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281b139e-c545-4213-ab8b-b83fafff473e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86\" (UID: \"281b139e-c545-4213-ab8b-b83fafff473e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:08.020742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.013947    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42781959-fe94-4542-aa6c-57f3b7a3e180-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8\" (UID: \"42781959-fe94-4542-aa6c-57f3b7a3e180\") " pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:08.020742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.014538    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281b139e-c545-4213-ab8b-b83fafff473e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86\" (UID: \"281b139e-c545-4213-ab8b-b83fafff473e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:08.043759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.033804    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm26v\" (UniqueName: \"kubernetes.io/projected/42781959-fe94-4542-aa6c-57f3b7a3e180-kube-api-access-wm26v\") pod \"payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8\" (UID: \"42781959-fe94-4542-aa6c-57f3b7a3e180\") " pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:08.043759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.042623    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm55j\" (UniqueName: \"kubernetes.io/projected/281b139e-c545-4213-ab8b-b83fafff473e-kube-api-access-pm55j\") pod \"payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86\" (UID: \"281b139e-c545-4213-ab8b-b83fafff473e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:08.116727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.116686    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:08.130967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.130936    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:08.474902 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:19:08.474867    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281b139e_c545_4213_ab8b_b83fafff473e.slice/crio-d7fcb7a405e6f08a377e78027a82c2f28534082117d913cc4163d1ac14581c8f WatchSource:0}: Error finding container d7fcb7a405e6f08a377e78027a82c2f28534082117d913cc4163d1ac14581c8f: Status 404 returned error can't find the container with id d7fcb7a405e6f08a377e78027a82c2f28534082117d913cc4163d1ac14581c8f
Oct 30 21:19:08.475044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.474974    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"]
Oct 30 21:19:08.646954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:08.646837    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"]
Oct 30 21:19:09.218600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:09.218511    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8" event={"ID":"42781959-fe94-4542-aa6c-57f3b7a3e180","Type":"ContainerStarted","Data":"4f2d6e964ab185cf0949d39427b6563820cb2845b5730c5cfb7fe68e3f09ea25"}
Oct 30 21:19:09.219114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:09.218707    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:09.219114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:09.218741    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8" event={"ID":"42781959-fe94-4542-aa6c-57f3b7a3e180","Type":"ContainerStarted","Data":"c79a34808c83ea3cc6f1016f9bd5cc7b1520cdb1f0991ae0bdd561383cc78204"}
Oct 30 21:19:09.220619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:09.220588    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86" event={"ID":"281b139e-c545-4213-ab8b-b83fafff473e","Type":"ContainerStarted","Data":"be113fba5d0c456f2dd88f41e1acc5940ae6dcb9cd225eac6808f8ad4bb4b585"}
Oct 30 21:19:09.220806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:09.220639    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86" event={"ID":"281b139e-c545-4213-ab8b-b83fafff473e","Type":"ContainerStarted","Data":"d7fcb7a405e6f08a377e78027a82c2f28534082117d913cc4163d1ac14581c8f"}
Oct 30 21:19:09.220806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:09.220707    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:09.240578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:09.240526    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8" podStartSLOduration=2.2405136 podStartE2EDuration="2.2405136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:09.235536376 +0000 UTC m=+1057.585263773" watchObservedRunningTime="2026-10-30 21:19:09.2405136 +0000 UTC m=+1057.590240995"
Oct 30 21:19:09.252222 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:09.252177    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86" podStartSLOduration=2.252165255 podStartE2EDuration="2.252165255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:09.249768857 +0000 UTC m=+1057.599496253" watchObservedRunningTime="2026-10-30 21:19:09.252165255 +0000 UTC m=+1057.601892651"
Oct 30 21:19:12.867333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:12.867293    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"]
Oct 30 21:19:12.877193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:12.876471    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:12.881742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:12.881716    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e494a9-serving-cert\""
Oct 30 21:19:12.886885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:12.886857    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"]
Oct 30 21:19:13.000813 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.000778    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmr5\" (UniqueName: \"kubernetes.io/projected/348cbb0a-d675-4025-bc1e-b13afd70dd5c-kube-api-access-grmr5\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.001013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.000844    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-metrics-certs\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.001013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.000932    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-maas-api-tls\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.001124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.001057    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348cbb0a-d675-4025-bc1e-b13afd70dd5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.106692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.105651    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmr5\" (UniqueName: \"kubernetes.io/projected/348cbb0a-d675-4025-bc1e-b13afd70dd5c-kube-api-access-grmr5\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.106692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.105784    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-metrics-certs\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.106692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.105849    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-maas-api-tls\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.106692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.105974    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348cbb0a-d675-4025-bc1e-b13afd70dd5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.110699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.108238    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348cbb0a-d675-4025-bc1e-b13afd70dd5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.129753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.125455    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-metrics-certs\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.129753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.126965    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-maas-api-tls\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.141039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.140950    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmr5\" (UniqueName: \"kubernetes.io/projected/348cbb0a-d675-4025-bc1e-b13afd70dd5c-kube-api-access-grmr5\") pod \"maas-api-e2e-api-b-e494a9-66f549c654-xk8m5\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.205552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.205519    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:13.596251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:13.596210    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"]
Oct 30 21:19:14.291461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:14.291418    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5" event={"ID":"348cbb0a-d675-4025-bc1e-b13afd70dd5c","Type":"ContainerStarted","Data":"72cb2f71eb94e82620e7f552beedaba0f512383b0dae6d45f185a378abcdbe5a"}
Oct 30 21:19:14.291953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:14.291501    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5" event={"ID":"348cbb0a-d675-4025-bc1e-b13afd70dd5c","Type":"ContainerStarted","Data":"7f1bfa76e8b45fdd83fde1504456c0bde2798a5fcda1169799191f961e2dff6e"}
Oct 30 21:19:14.291953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:14.291530    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:14.313320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:14.313274    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5" podStartSLOduration=1.81355507 podStartE2EDuration="2.313259821s" totalImagesPullingTime="499.704751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:14.309993307 +0000 UTC m=+1062.659720733" watchObservedRunningTime="2026-10-30 21:19:14.313259821 +0000 UTC m=+1062.662987216"
Oct 30 21:19:20.232011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:20.231979    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:19:20.232526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:20.232492    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:19:20.305990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:20.305958    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:19:21.608584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.608549    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"]
Oct 30 21:19:21.612984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.612964    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:21.617769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.617652    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4c6998\""
Oct 30 21:19:21.618460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.618437    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4c6998-dockercfg-bd7fq\""
Oct 30 21:19:21.626710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.624858    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"]
Oct 30 21:19:21.738712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.738611    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh\" (UID: \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:21.739124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.739102    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsss7\" (UniqueName: \"kubernetes.io/projected/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-kube-api-access-vsss7\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh\" (UID: \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:21.885057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.884959    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh\" (UID: \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:21.922118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.922081    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsss7\" (UniqueName: \"kubernetes.io/projected/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-kube-api-access-vsss7\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh\" (UID: \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:21.922324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.889321    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh\" (UID: \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:21.935716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:21.933920    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsss7\" (UniqueName: \"kubernetes.io/projected/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-kube-api-access-vsss7\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh\" (UID: \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:22.230620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:22.230590    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:22.524372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:22.524338    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"]
Oct 30 21:19:22.524544 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:19:22.524515    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd5ca82_d8ca_46ad_b777_8025f0c3165e.slice/crio-bb77519bd08b5423dfd493868db4379ff022548295399402806053509f7d3f7d WatchSource:0}: Error finding container bb77519bd08b5423dfd493868db4379ff022548295399402806053509f7d3f7d: Status 404 returned error can't find the container with id bb77519bd08b5423dfd493868db4379ff022548295399402806053509f7d3f7d
Oct 30 21:19:23.437693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:23.434452    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh" event={"ID":"2dd5ca82-d8ca-46ad-b777-8025f0c3165e","Type":"ContainerStarted","Data":"50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121"}
Oct 30 21:19:23.437693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:23.434563    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh" event={"ID":"2dd5ca82-d8ca-46ad-b777-8025f0c3165e","Type":"ContainerStarted","Data":"bb77519bd08b5423dfd493868db4379ff022548295399402806053509f7d3f7d"}
Oct 30 21:19:23.462694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:23.439396    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:23.533708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:23.533623    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh" podStartSLOduration=2.5336000480000003 podStartE2EDuration="2.533600048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:23.497190017 +0000 UTC m=+1071.846917406" watchObservedRunningTime="2026-10-30 21:19:23.533600048 +0000 UTC m=+1071.883327447"
Oct 30 21:19:25.255696 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:25.255275    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:19:26.521694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:26.514104    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"]
Oct 30 21:19:26.805006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:26.804914    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"]
Oct 30 21:19:26.805351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:26.805328    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:26.861694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:26.854930    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5bc8c60a-serving-cert\""
Oct 30 21:19:26.861694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:26.855109    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca5e5fd-ae1a-4243-b63b-101685b87c12-openshift-service-ca\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:26.861694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:26.855313    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:26.861694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:26.855399    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r62n9\" (UniqueName: \"kubernetes.io/projected/3ca5e5fd-ae1a-4243-b63b-101685b87c12-kube-api-access-r62n9\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:26.861694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:26.855562    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-metrics-certs\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.036025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.027542    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.036390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.036358    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r62n9\" (UniqueName: \"kubernetes.io/projected/3ca5e5fd-ae1a-4243-b63b-101685b87c12-kube-api-access-r62n9\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.059047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.036762    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-metrics-certs\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.070695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.062441    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca5e5fd-ae1a-4243-b63b-101685b87c12-openshift-service-ca\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.070695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.063513    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca5e5fd-ae1a-4243-b63b-101685b87c12-openshift-service-ca\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.070695 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:27.033454    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-serving-cert: secret "maas-api-e2e-mt-5bc8c60a-serving-cert" not found
Oct 30 21:19:27.070695 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:27.068040    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls podName:3ca5e5fd-ae1a-4243-b63b-101685b87c12 nodeName:}" failed. No retries permitted until 2026-10-30 21:19:27.567995564 +0000 UTC m=+1075.917722951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls") pod "maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt" (UID: "3ca5e5fd-ae1a-4243-b63b-101685b87c12") : secret "maas-api-e2e-mt-5bc8c60a-serving-cert" not found
Oct 30 21:19:27.507560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.507514    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-metrics-certs\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.508603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.508573    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r62n9\" (UniqueName: \"kubernetes.io/projected/3ca5e5fd-ae1a-4243-b63b-101685b87c12-kube-api-access-r62n9\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.653350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.636893    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.653350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.653257    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls\") pod \"maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:27.781034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:27.780935    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:28.530543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:28.530466    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"]
Oct 30 21:19:28.536680 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:19:28.536622    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca5e5fd_ae1a_4243_b63b_101685b87c12.slice/crio-7d7c246f1ec70f57675b9c87c29808c3e6d9f89c67dda751c3a54c0e8effa2bf WatchSource:0}: Error finding container 7d7c246f1ec70f57675b9c87c29808c3e6d9f89c67dda751c3a54c0e8effa2bf: Status 404 returned error can't find the container with id 7d7c246f1ec70f57675b9c87c29808c3e6d9f89c67dda751c3a54c0e8effa2bf
Oct 30 21:19:28.996785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:28.996749    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt" event={"ID":"3ca5e5fd-ae1a-4243-b63b-101685b87c12","Type":"ContainerStarted","Data":"7d7c246f1ec70f57675b9c87c29808c3e6d9f89c67dda751c3a54c0e8effa2bf"}
Oct 30 21:19:30.007141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.007102    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt" event={"ID":"3ca5e5fd-ae1a-4243-b63b-101685b87c12","Type":"ContainerStarted","Data":"b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65"}
Oct 30 21:19:30.007572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.007174    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:30.031449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.031383    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt" podStartSLOduration=3.631956573 podStartE2EDuration="4.031363529s" totalImagesPullingTime="399.406956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:30.030391776 +0000 UTC m=+1078.380119255" watchObservedRunningTime="2026-10-30 21:19:30.031363529 +0000 UTC m=+1078.381090926"
Oct 30 21:19:30.219280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.219243    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"]
Oct 30 21:19:30.223913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.223894    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:30.230794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.230769    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"]
Oct 30 21:19:30.295050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.294790    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3ac6bf\""
Oct 30 21:19:30.295418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.295370    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3ac6bf-dockercfg-vnxdv\""
Oct 30 21:19:30.352841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.350144    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzg4q\" (UniqueName: \"kubernetes.io/projected/e76bf41a-179a-41a0-9e47-2a2f306b3d13-kube-api-access-vzg4q\") pod \"payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v\" (UID: \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:30.352841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.350580    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76bf41a-179a-41a0-9e47-2a2f306b3d13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v\" (UID: \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:30.469701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.467611    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzg4q\" (UniqueName: \"kubernetes.io/projected/e76bf41a-179a-41a0-9e47-2a2f306b3d13-kube-api-access-vzg4q\") pod \"payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v\" (UID: \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:30.469701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.467947    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76bf41a-179a-41a0-9e47-2a2f306b3d13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v\" (UID: \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:30.469701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.468927    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76bf41a-179a-41a0-9e47-2a2f306b3d13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v\" (UID: \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:30.511748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.511569    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzg4q\" (UniqueName: \"kubernetes.io/projected/e76bf41a-179a-41a0-9e47-2a2f306b3d13-kube-api-access-vzg4q\") pod \"payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v\" (UID: \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:30.541442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.541413    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:30.572326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.572290    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"]
Oct 30 21:19:30.588746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.586894    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"]
Oct 30 21:19:30.588746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.587013    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.593065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.592450    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4c6998-serving-cert\""
Oct 30 21:19:30.671409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.671376    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-metrics-certs\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.671577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.671435    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.671704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.671647    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhggs\" (UniqueName: \"kubernetes.io/projected/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-kube-api-access-dhggs\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.671784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.671763    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.778208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.773068    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.778208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.773289    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-metrics-certs\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.778208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.773329    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.778208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.773465    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhggs\" (UniqueName: \"kubernetes.io/projected/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-kube-api-access-dhggs\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.778208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.775495    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.778208 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:30.775655    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-serving-cert: secret "maas-api-e2e-ipp-a-4c6998-serving-cert" not found
Oct 30 21:19:30.778208 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:30.775740    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls podName:e4001d79-f0a5-4dbe-bf55-d8ec626b2c40 nodeName:}" failed. No retries permitted until 2026-10-30 21:19:31.275719885 +0000 UTC m=+1079.625447276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls") pod "maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh" (UID: "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40") : secret "maas-api-e2e-ipp-a-4c6998-serving-cert" not found
Oct 30 21:19:30.785307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.781924    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-metrics-certs\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.787729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.787703    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhggs\" (UniqueName: \"kubernetes.io/projected/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-kube-api-access-dhggs\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:30.876230 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:19:30.876205    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76bf41a_179a_41a0_9e47_2a2f306b3d13.slice/crio-97e6693b3ae21bd5a58d747f96745926dc0ad300ec4189bfcb4c2c980cd97348 WatchSource:0}: Error finding container 97e6693b3ae21bd5a58d747f96745926dc0ad300ec4189bfcb4c2c980cd97348: Status 404 returned error can't find the container with id 97e6693b3ae21bd5a58d747f96745926dc0ad300ec4189bfcb4c2c980cd97348
Oct 30 21:19:30.878813 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:30.878782    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"]
Oct 30 21:19:31.018472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:31.017737    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v" event={"ID":"e76bf41a-179a-41a0-9e47-2a2f306b3d13","Type":"ContainerStarted","Data":"f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8"}
Oct 30 21:19:31.018472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:31.017800    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v" event={"ID":"e76bf41a-179a-41a0-9e47-2a2f306b3d13","Type":"ContainerStarted","Data":"97e6693b3ae21bd5a58d747f96745926dc0ad300ec4189bfcb4c2c980cd97348"}
Oct 30 21:19:31.018472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:31.018037    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:31.045231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:31.045167    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v" podStartSLOduration=1.045147366 podStartE2EDuration="1.045147366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:31.040640895 +0000 UTC m=+1079.390368291" watchObservedRunningTime="2026-10-30 21:19:31.045147366 +0000 UTC m=+1079.394874762"
Oct 30 21:19:31.286712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:31.284906    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:31.298704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:31.295339    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:31.519386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:31.519342    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:31.822255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:31.822226    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"]
Oct 30 21:19:31.822725 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:19:31.822695    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4001d79_f0a5_4dbe_bf55_d8ec626b2c40.slice/crio-93fba251859722b1d7066ca5c39793f8157c0f803135597eeced9813aa1a113e WatchSource:0}: Error finding container 93fba251859722b1d7066ca5c39793f8157c0f803135597eeced9813aa1a113e: Status 404 returned error can't find the container with id 93fba251859722b1d7066ca5c39793f8157c0f803135597eeced9813aa1a113e
Oct 30 21:19:32.028212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:32.028172    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh" event={"ID":"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40","Type":"ContainerStarted","Data":"93fba251859722b1d7066ca5c39793f8157c0f803135597eeced9813aa1a113e"}
Oct 30 21:19:33.037986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:33.037948    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh" event={"ID":"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40","Type":"ContainerStarted","Data":"bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4"}
Oct 30 21:19:33.038374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:33.038014    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:33.058695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:33.058616    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh" podStartSLOduration=2.568002712 podStartE2EDuration="3.058596676s" totalImagesPullingTime="490.593964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:33.054238129 +0000 UTC m=+1081.403965525" watchObservedRunningTime="2026-10-30 21:19:33.058596676 +0000 UTC m=+1081.408324076"
Oct 30 21:19:35.498459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:35.498426    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:36.024874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.024844    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:19:36.871445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.871415    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"]
Oct 30 21:19:36.876166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.876146    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:36.879686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.879646    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3ac6bf-serving-cert\""
Oct 30 21:19:36.884805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.884784    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"]
Oct 30 21:19:36.982060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.982028    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:36.982257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.982071    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:36.982316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.982249    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsb4p\" (UniqueName: \"kubernetes.io/projected/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-kube-api-access-qsb4p\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:36.982449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:36.982431    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.122120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.122000    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.122608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.122578    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.122816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.122628    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.122883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.122810    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsb4p\" (UniqueName: \"kubernetes.io/projected/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-kube-api-access-qsb4p\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.131450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.131415    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.142852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.142492    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsb4p\" (UniqueName: \"kubernetes.io/projected/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-kube-api-access-qsb4p\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.142852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.142784    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.142852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.142792    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.197821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.197792    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:37.314309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.314274    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"]
Oct 30 21:19:37.493117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:37.493084    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"]
Oct 30 21:19:37.493836 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:19:37.493727    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4074b6d4_ca47_47ba_9677_0e0b9ba2099d.slice/crio-011ccb1e857932fcfe4fcb3ac7dd6b54e79e2b7a62d23338ac6d956c018e7f26 WatchSource:0}: Error finding container 011ccb1e857932fcfe4fcb3ac7dd6b54e79e2b7a62d23338ac6d956c018e7f26: Status 404 returned error can't find the container with id 011ccb1e857932fcfe4fcb3ac7dd6b54e79e2b7a62d23338ac6d956c018e7f26
Oct 30 21:19:38.131310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:38.131273    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5" event={"ID":"4074b6d4-ca47-47ba-9677-0e0b9ba2099d","Type":"ContainerStarted","Data":"6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5"}
Oct 30 21:19:38.131775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:38.131341    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5" event={"ID":"4074b6d4-ca47-47ba-9677-0e0b9ba2099d","Type":"ContainerStarted","Data":"011ccb1e857932fcfe4fcb3ac7dd6b54e79e2b7a62d23338ac6d956c018e7f26"}
Oct 30 21:19:38.131775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:38.131488    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:38.187690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:38.183753    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5" podStartSLOduration=1.7388845179999999 podStartE2EDuration="2.183728229s" totalImagesPullingTime="444.843711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:38.164771173 +0000 UTC m=+1086.514498568" watchObservedRunningTime="2026-10-30 21:19:38.183728229 +0000 UTC m=+1086.533455626"
Oct 30 21:19:39.051284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:39.051256    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:19:42.040817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.040789    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:42.318876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.318779    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd" podUID="f590b8fa-db79-49f5-b40d-5386e2694fa9" containerName="payload-processing" containerID="cri-o://f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df" gracePeriod=25
Oct 30 21:19:42.668709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.664939    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:19:42.774578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.774540    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f590b8fa-db79-49f5-b40d-5386e2694fa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f590b8fa-db79-49f5-b40d-5386e2694fa9-plugins-config-volume\") pod \"f590b8fa-db79-49f5-b40d-5386e2694fa9\" (UID: \"f590b8fa-db79-49f5-b40d-5386e2694fa9\") "
Oct 30 21:19:42.774770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.774752    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f590b8fa-db79-49f5-b40d-5386e2694fa9-kube-api-access-8vk7r\" (UniqueName: \"kubernetes.io/projected/f590b8fa-db79-49f5-b40d-5386e2694fa9-kube-api-access-8vk7r\") pod \"f590b8fa-db79-49f5-b40d-5386e2694fa9\" (UID: \"f590b8fa-db79-49f5-b40d-5386e2694fa9\") "
Oct 30 21:19:42.778343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.778117    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f590b8fa-db79-49f5-b40d-5386e2694fa9-plugins-config-volume" pod "f590b8fa-db79-49f5-b40d-5386e2694fa9" (UID: "f590b8fa-db79-49f5-b40d-5386e2694fa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:42.778465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.778429    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f590b8fa-db79-49f5-b40d-5386e2694fa9-kube-api-access-8vk7r" pod "f590b8fa-db79-49f5-b40d-5386e2694fa9" (UID: "f590b8fa-db79-49f5-b40d-5386e2694fa9"). InnerVolumeSpecName "kube-api-access-8vk7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:42.876502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.876426    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f590b8fa-db79-49f5-b40d-5386e2694fa9-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:42.876502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:42.876454    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vk7r\" (UniqueName: \"kubernetes.io/projected/f590b8fa-db79-49f5-b40d-5386e2694fa9-kube-api-access-8vk7r\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:43.277454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.277419    4633 generic.go:356] "Generic (PLEG): container finished" podID="f590b8fa-db79-49f5-b40d-5386e2694fa9" containerID="f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df" exitCode=0
Oct 30 21:19:43.277897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.277461    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"
Oct 30 21:19:43.277897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.277506    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd" event={"ID":"f590b8fa-db79-49f5-b40d-5386e2694fa9","Type":"ContainerDied","Data":"f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df"}
Oct 30 21:19:43.277897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.277555    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd" event={"ID":"f590b8fa-db79-49f5-b40d-5386e2694fa9","Type":"ContainerDied","Data":"bf9e59e0226e6e79296d86b82824c6bf8d7c21612a591626e19f6f665bdffc74"}
Oct 30 21:19:43.277897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.277580    4633 scope.go:122] "RemoveContainer" containerID="f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df"
Oct 30 21:19:43.289325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.289312    4633 scope.go:122] "RemoveContainer" containerID="f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df"
Oct 30 21:19:43.289580 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:43.289564    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df\": container with ID starting with f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df not found: ID does not exist" containerID="f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df"
Oct 30 21:19:43.289661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.289590    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df"} err="failed to get container status \"f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df\": rpc error: code = NotFound desc = could not find container \"f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df\": container with ID starting with f5a9af0870e2b8b43e499be0f0865ecc162c3afff13229b2e5e3bcbe134ae2df not found: ID does not exist"
Oct 30 21:19:43.300614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.300593    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"]
Oct 30 21:19:43.304652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.304630    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4aac-648d9bf6dc-rsznd"]
Oct 30 21:19:43.796741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.794274    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f590b8fa-db79-49f5-b40d-5386e2694fa9" path="/var/lib/kubelet/pods/f590b8fa-db79-49f5-b40d-5386e2694fa9/volumes"
Oct 30 21:19:43.813716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:43.812775    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:19:44.166627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:44.166456    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:19:45.482699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:45.482642    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"]
Oct 30 21:19:48.935196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:48.935158    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"]
Oct 30 21:19:50.433633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.433604    4633 generic.go:356] "Generic (PLEG): container finished" podID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerID="405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff" exitCode=0
Oct 30 21:19:50.434140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.433693    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" event={"ID":"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f","Type":"ContainerDied","Data":"405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff"}
Oct 30 21:19:50.434140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.434138    4633 scope.go:122] "RemoveContainer" containerID="405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff"
Oct 30 21:19:50.435777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.435748    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:19:50.435881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.435823    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:19:50.435881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.435856    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:19:50.488055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.488016    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh" podUID="2dd5ca82-d8ca-46ad-b777-8025f0c3165e" containerName="payload-pre-processing" containerID="cri-o://50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121" gracePeriod=25
Oct 30 21:19:50.782753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.782724    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:50.945060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.945022    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-kube-api-access-vsss7\" (UniqueName: \"kubernetes.io/projected/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-kube-api-access-vsss7\") pod \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\" (UID: \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\") "
Oct 30 21:19:50.945251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.945198    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-plugins-config-volume\") pod \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\" (UID: \"2dd5ca82-d8ca-46ad-b777-8025f0c3165e\") "
Oct 30 21:19:50.945660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.945611    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-plugins-config-volume" pod "2dd5ca82-d8ca-46ad-b777-8025f0c3165e" (UID: "2dd5ca82-d8ca-46ad-b777-8025f0c3165e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:50.947948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.947924    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:50.948059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:50.948035    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-kube-api-access-vsss7" pod "2dd5ca82-d8ca-46ad-b777-8025f0c3165e" (UID: "2dd5ca82-d8ca-46ad-b777-8025f0c3165e"). InnerVolumeSpecName "kube-api-access-vsss7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:51.053005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.050041    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsss7\" (UniqueName: \"kubernetes.io/projected/2dd5ca82-d8ca-46ad-b777-8025f0c3165e-kube-api-access-vsss7\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:51.449267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.449228    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" event={"ID":"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f","Type":"ContainerStarted","Data":"a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90"}
Oct 30 21:19:51.450738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.450713    4633 generic.go:356] "Generic (PLEG): container finished" podID="2dd5ca82-d8ca-46ad-b777-8025f0c3165e" containerID="50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121" exitCode=0
Oct 30 21:19:51.450864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.450755    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"
Oct 30 21:19:51.450864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.450763    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh" event={"ID":"2dd5ca82-d8ca-46ad-b777-8025f0c3165e","Type":"ContainerDied","Data":"50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121"}
Oct 30 21:19:51.450864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.450811    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh" event={"ID":"2dd5ca82-d8ca-46ad-b777-8025f0c3165e","Type":"ContainerDied","Data":"bb77519bd08b5423dfd493868db4379ff022548295399402806053509f7d3f7d"}
Oct 30 21:19:51.450864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.450834    4633 scope.go:122] "RemoveContainer" containerID="50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121"
Oct 30 21:19:51.462480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.462463    4633 scope.go:122] "RemoveContainer" containerID="50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121"
Oct 30 21:19:51.462766 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:51.462746    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121\": container with ID starting with 50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121 not found: ID does not exist" containerID="50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121"
Oct 30 21:19:51.462845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.462778    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121"} err="failed to get container status \"50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121\": rpc error: code = NotFound desc = could not find container \"50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121\": container with ID starting with 50fa7a0e73856bbec43d66f717167e00857c737bc907c716eff90ecf5e5af121 not found: ID does not exist"
Oct 30 21:19:51.488835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.488811    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"]
Oct 30 21:19:51.492894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.492871    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-9cc7b675-rr2sh"]
Oct 30 21:19:51.499044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.499022    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:19:51.500187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.500164    4633 patch_prober.go:29] interesting pod/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 21:19:51.500278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.500206    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 21:19:51.794661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:51.794589    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd5ca82-d8ca-46ad-b777-8025f0c3165e" path="/var/lib/kubelet/pods/2dd5ca82-d8ca-46ad-b777-8025f0c3165e/volumes"
Oct 30 21:19:52.499325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:52.499295    4633 patch_prober.go:29] interesting pod/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 21:19:52.499750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:52.499350    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 21:19:53.498980 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:53.498946    4633 patch_prober.go:29] interesting pod/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 21:19:53.499164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:53.499020    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 21:19:53.937732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:53.935744    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v" podUID="e76bf41a-179a-41a0-9e47-2a2f306b3d13" containerName="payload-processing" containerID="cri-o://f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8" gracePeriod=25
Oct 30 21:19:54.340885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.340847    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:19:54.340885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.340881    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:54.445863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.445827    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76bf41a-179a-41a0-9e47-2a2f306b3d13-kube-api-access-vzg4q\" (UniqueName: \"kubernetes.io/projected/e76bf41a-179a-41a0-9e47-2a2f306b3d13-kube-api-access-vzg4q\") pod \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\" (UID: \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\") "
Oct 30 21:19:54.446073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.446055    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76bf41a-179a-41a0-9e47-2a2f306b3d13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76bf41a-179a-41a0-9e47-2a2f306b3d13-plugins-config-volume\") pod \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\" (UID: \"e76bf41a-179a-41a0-9e47-2a2f306b3d13\") "
Oct 30 21:19:54.446437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.446409    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76bf41a-179a-41a0-9e47-2a2f306b3d13-plugins-config-volume" pod "e76bf41a-179a-41a0-9e47-2a2f306b3d13" (UID: "e76bf41a-179a-41a0-9e47-2a2f306b3d13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:54.446886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.446870    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76bf41a-179a-41a0-9e47-2a2f306b3d13-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:54.448309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.448282    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76bf41a-179a-41a0-9e47-2a2f306b3d13-kube-api-access-vzg4q" pod "e76bf41a-179a-41a0-9e47-2a2f306b3d13" (UID: "e76bf41a-179a-41a0-9e47-2a2f306b3d13"). InnerVolumeSpecName "kube-api-access-vzg4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:54.481897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.481867    4633 generic.go:356] "Generic (PLEG): container finished" podID="e76bf41a-179a-41a0-9e47-2a2f306b3d13" containerID="f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8" exitCode=0
Oct 30 21:19:54.482035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.481914    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"
Oct 30 21:19:54.482035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.481946    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v" event={"ID":"e76bf41a-179a-41a0-9e47-2a2f306b3d13","Type":"ContainerDied","Data":"f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8"}
Oct 30 21:19:54.482035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.482002    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v" event={"ID":"e76bf41a-179a-41a0-9e47-2a2f306b3d13","Type":"ContainerDied","Data":"97e6693b3ae21bd5a58d747f96745926dc0ad300ec4189bfcb4c2c980cd97348"}
Oct 30 21:19:54.482035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.482026    4633 scope.go:122] "RemoveContainer" containerID="f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8"
Oct 30 21:19:54.492969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.492952    4633 scope.go:122] "RemoveContainer" containerID="f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8"
Oct 30 21:19:54.493257 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:54.493238    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8\": container with ID starting with f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8 not found: ID does not exist" containerID="f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8"
Oct 30 21:19:54.493305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.493266    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8"} err="failed to get container status \"f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8\": rpc error: code = NotFound desc = could not find container \"f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8\": container with ID starting with f4077b6cf4fa0e48b3c6ec8aead5c89602d0b4dd45ca5d6299c53828208dc3b8 not found: ID does not exist"
Oct 30 21:19:54.499617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.499596    4633 patch_prober.go:29] interesting pod/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 21:19:54.499721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.499640    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 21:19:54.545733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.545560    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"]
Oct 30 21:19:54.564965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.555127    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzg4q\" (UniqueName: \"kubernetes.io/projected/e76bf41a-179a-41a0-9e47-2a2f306b3d13-kube-api-access-vzg4q\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:54.590436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.583932    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-98db7966c-z4v8v"]
Oct 30 21:19:54.826751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.825794    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"]
Oct 30 21:19:54.826751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:54.826034    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs" podUID="ce3f7242-71a0-410f-8923-c4fdea2c393c" containerName="maas-api" containerID="cri-o://7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904" gracePeriod=30
Oct 30 21:19:55.091570 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:55.091489    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:19:55.471359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.471331    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:19:55.496112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.492899    4633 generic.go:356] "Generic (PLEG): container finished" podID="ce3f7242-71a0-410f-8923-c4fdea2c393c" containerID="7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904" exitCode=0
Oct 30 21:19:55.496112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.492956    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"
Oct 30 21:19:55.496112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.493024    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs" event={"ID":"ce3f7242-71a0-410f-8923-c4fdea2c393c","Type":"ContainerDied","Data":"7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904"}
Oct 30 21:19:55.496112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.493066    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs" event={"ID":"ce3f7242-71a0-410f-8923-c4fdea2c393c","Type":"ContainerDied","Data":"d133908e29be0593dfaa099514198f8b9427ea4c621c36172b22b8b244272dd7"}
Oct 30 21:19:55.496112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.493096    4633 scope.go:122] "RemoveContainer" containerID="7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904"
Oct 30 21:19:55.500498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.500471    4633 patch_prober.go:29] interesting pod/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 21:19:55.500732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.500528    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 21:19:55.506852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.506826    4633 scope.go:122] "RemoveContainer" containerID="7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904"
Oct 30 21:19:55.507268 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:19:55.507239    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904\": container with ID starting with 7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904 not found: ID does not exist" containerID="7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904"
Oct 30 21:19:55.507358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.507280    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904"} err="failed to get container status \"7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904\": rpc error: code = NotFound desc = could not find container \"7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904\": container with ID starting with 7cd4266c0f8ad240799ea9396e1b247972a230f063bf5f832dbe96bfb901a904 not found: ID does not exist"
Oct 30 21:19:55.572790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.572757    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3f7242-71a0-410f-8923-c4fdea2c393c-kube-api-access-pxqf2\" (UniqueName: \"kubernetes.io/projected/ce3f7242-71a0-410f-8923-c4fdea2c393c-kube-api-access-pxqf2\") pod \"ce3f7242-71a0-410f-8923-c4fdea2c393c\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") "
Oct 30 21:19:55.572961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.572886    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-maas-api-tls\") pod \"ce3f7242-71a0-410f-8923-c4fdea2c393c\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") "
Oct 30 21:19:55.573018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.573004    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-metrics-certs\") pod \"ce3f7242-71a0-410f-8923-c4fdea2c393c\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") "
Oct 30 21:19:55.573139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.573078    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3f7242-71a0-410f-8923-c4fdea2c393c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce3f7242-71a0-410f-8923-c4fdea2c393c-openshift-service-ca\") pod \"ce3f7242-71a0-410f-8923-c4fdea2c393c\" (UID: \"ce3f7242-71a0-410f-8923-c4fdea2c393c\") "
Oct 30 21:19:55.573803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.573632    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3f7242-71a0-410f-8923-c4fdea2c393c-openshift-service-ca" pod "ce3f7242-71a0-410f-8923-c4fdea2c393c" (UID: "ce3f7242-71a0-410f-8923-c4fdea2c393c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:55.575408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.575375    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3f7242-71a0-410f-8923-c4fdea2c393c-kube-api-access-pxqf2" pod "ce3f7242-71a0-410f-8923-c4fdea2c393c" (UID: "ce3f7242-71a0-410f-8923-c4fdea2c393c"). InnerVolumeSpecName "kube-api-access-pxqf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:55.575792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.575772    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-metrics-certs" pod "ce3f7242-71a0-410f-8923-c4fdea2c393c" (UID: "ce3f7242-71a0-410f-8923-c4fdea2c393c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:55.575867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.575808    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-maas-api-tls" pod "ce3f7242-71a0-410f-8923-c4fdea2c393c" (UID: "ce3f7242-71a0-410f-8923-c4fdea2c393c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:55.675021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.674982    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:55.675021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.675019    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce3f7242-71a0-410f-8923-c4fdea2c393c-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:55.675244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.675034    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce3f7242-71a0-410f-8923-c4fdea2c393c-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:55.675244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.675043    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqf2\" (UniqueName: \"kubernetes.io/projected/ce3f7242-71a0-410f-8923-c4fdea2c393c-kube-api-access-pxqf2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:55.792236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.792159    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76bf41a-179a-41a0-9e47-2a2f306b3d13" path="/var/lib/kubelet/pods/e76bf41a-179a-41a0-9e47-2a2f306b3d13/volumes"
Oct 30 21:19:55.845530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.845479    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"]
Oct 30 21:19:55.849496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:55.849480    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0c4aac-5f94499dc7-6vsbs"]
Oct 30 21:19:56.498869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:56.498831    4633 patch_prober.go:29] interesting pod/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 30 21:19:56.499286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:56.498926    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 30 21:19:57.536276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:57.536245    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:19:57.537639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:57.537616    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:19:57.847056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:19:57.846954    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3f7242-71a0-410f-8923-c4fdea2c393c" path="/var/lib/kubelet/pods/ce3f7242-71a0-410f-8923-c4fdea2c393c/volumes"
Oct 30 21:20:00.930337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:00.930304    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"]
Oct 30 21:20:00.930850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:00.930564    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt" podUID="3ca5e5fd-ae1a-4243-b63b-101685b87c12" containerName="maas-api" containerID="cri-o://b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65" gracePeriod=30
Oct 30 21:20:01.018624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.018587    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt" podUID="3ca5e5fd-ae1a-4243-b63b-101685b87c12" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.123:8443/health\": dial tcp 10.133.0.123:8443: connect: connection refused"
Oct 30 21:20:01.472232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.472202    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:20:01.506132 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.506102    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca5e5fd-ae1a-4243-b63b-101685b87c12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca5e5fd-ae1a-4243-b63b-101685b87c12-openshift-service-ca\") pod \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") "
Oct 30 21:20:01.506311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.506242    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-metrics-certs\") pod \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") "
Oct 30 21:20:01.506311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.506298    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls\") pod \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") "
Oct 30 21:20:01.506433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.506354    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca5e5fd-ae1a-4243-b63b-101685b87c12-kube-api-access-r62n9\" (UniqueName: \"kubernetes.io/projected/3ca5e5fd-ae1a-4243-b63b-101685b87c12-kube-api-access-r62n9\") pod \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\" (UID: \"3ca5e5fd-ae1a-4243-b63b-101685b87c12\") "
Oct 30 21:20:01.506494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.506466    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca5e5fd-ae1a-4243-b63b-101685b87c12-openshift-service-ca" pod "3ca5e5fd-ae1a-4243-b63b-101685b87c12" (UID: "3ca5e5fd-ae1a-4243-b63b-101685b87c12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:01.506908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.506887    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca5e5fd-ae1a-4243-b63b-101685b87c12-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:01.508698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.508651    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-metrics-certs" pod "3ca5e5fd-ae1a-4243-b63b-101685b87c12" (UID: "3ca5e5fd-ae1a-4243-b63b-101685b87c12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:01.508801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.508695    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls" pod "3ca5e5fd-ae1a-4243-b63b-101685b87c12" (UID: "3ca5e5fd-ae1a-4243-b63b-101685b87c12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:01.508801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.508708    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca5e5fd-ae1a-4243-b63b-101685b87c12-kube-api-access-r62n9" pod "3ca5e5fd-ae1a-4243-b63b-101685b87c12" (UID: "3ca5e5fd-ae1a-4243-b63b-101685b87c12"). InnerVolumeSpecName "kube-api-access-r62n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:01.608036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.607990    4633 generic.go:356] "Generic (PLEG): container finished" podID="3ca5e5fd-ae1a-4243-b63b-101685b87c12" containerID="b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65" exitCode=0
Oct 30 21:20:01.608229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.608062    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:01.608229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.608082    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ca5e5fd-ae1a-4243-b63b-101685b87c12-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:01.608229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.608094    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r62n9\" (UniqueName: \"kubernetes.io/projected/3ca5e5fd-ae1a-4243-b63b-101685b87c12-kube-api-access-r62n9\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:01.608229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.608095    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt" event={"ID":"3ca5e5fd-ae1a-4243-b63b-101685b87c12","Type":"ContainerDied","Data":"b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65"}
Oct 30 21:20:01.608229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.608121    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"
Oct 30 21:20:01.608229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.608186    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt" event={"ID":"3ca5e5fd-ae1a-4243-b63b-101685b87c12","Type":"ContainerDied","Data":"7d7c246f1ec70f57675b9c87c29808c3e6d9f89c67dda751c3a54c0e8effa2bf"}
Oct 30 21:20:01.608229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.608219    4633 scope.go:122] "RemoveContainer" containerID="b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65"
Oct 30 21:20:01.623091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.623072    4633 scope.go:122] "RemoveContainer" containerID="b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65"
Oct 30 21:20:01.623616 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:01.623590    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65\": container with ID starting with b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65 not found: ID does not exist" containerID="b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65"
Oct 30 21:20:01.623764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.623622    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65"} err="failed to get container status \"b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65\": rpc error: code = NotFound desc = could not find container \"b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65\": container with ID starting with b22220a91cf090e8f5b5f8f0a899b93401b293394f96420c0cbd58a10ffb5d65 not found: ID does not exist"
Oct 30 21:20:01.633550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.633530    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"]
Oct 30 21:20:01.642509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.642479    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5bc8c60a-6cd9986bdb-rqlpt"]
Oct 30 21:20:01.794911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:01.794835    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca5e5fd-ae1a-4243-b63b-101685b87c12" path="/var/lib/kubelet/pods/3ca5e5fd-ae1a-4243-b63b-101685b87c12/volumes"
Oct 30 21:20:02.817133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:02.817094    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"]
Oct 30 21:20:02.817627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:02.817342    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy" containerID="cri-o://a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90" gracePeriod=30
Oct 30 21:20:07.826574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.826539    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"]
Oct 30 21:20:07.829683 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829631    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f590b8fa-db79-49f5-b40d-5386e2694fa9" containerName="payload-processing"
Oct 30 21:20:07.829822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829685    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f590b8fa-db79-49f5-b40d-5386e2694fa9" containerName="payload-processing"
Oct 30 21:20:07.829822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829705    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2dd5ca82-d8ca-46ad-b777-8025f0c3165e" containerName="payload-pre-processing"
Oct 30 21:20:07.829822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829713    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd5ca82-d8ca-46ad-b777-8025f0c3165e" containerName="payload-pre-processing"
Oct 30 21:20:07.829822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829758    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e76bf41a-179a-41a0-9e47-2a2f306b3d13" containerName="payload-processing"
Oct 30 21:20:07.829822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829766    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76bf41a-179a-41a0-9e47-2a2f306b3d13" containerName="payload-processing"
Oct 30 21:20:07.829822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829796    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ce3f7242-71a0-410f-8923-c4fdea2c393c" containerName="maas-api"
Oct 30 21:20:07.829822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829804    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3f7242-71a0-410f-8923-c4fdea2c393c" containerName="maas-api"
Oct 30 21:20:07.830221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829846    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3ca5e5fd-ae1a-4243-b63b-101685b87c12" containerName="maas-api"
Oct 30 21:20:07.830221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.829854    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca5e5fd-ae1a-4243-b63b-101685b87c12" containerName="maas-api"
Oct 30 21:20:07.830221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.830019    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76bf41a-179a-41a0-9e47-2a2f306b3d13" containerName="payload-processing"
Oct 30 21:20:07.830221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.830039    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3f7242-71a0-410f-8923-c4fdea2c393c" containerName="maas-api"
Oct 30 21:20:07.830221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.830055    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca5e5fd-ae1a-4243-b63b-101685b87c12" containerName="maas-api"
Oct 30 21:20:07.830221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.830066    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f590b8fa-db79-49f5-b40d-5386e2694fa9" containerName="payload-processing"
Oct 30 21:20:07.830221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.830075    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd5ca82-d8ca-46ad-b777-8025f0c3165e" containerName="payload-pre-processing"
Oct 30 21:20:07.849999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.847165    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:07.853644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.853614    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"]
Oct 30 21:20:07.951701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:07.945762    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-43f6ab-openshift-default-dockercfg-k27dw\""
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.026769    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-credential-socket\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.026932    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f416feb-13be-4a2c-b2bc-9999484d1d56-istiod-ca-cert\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.027029    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-certs\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.027054    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-podinfo\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.027204    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-data\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.027249    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-socket\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.027298    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vz8n\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-kube-api-access-7vz8n\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.027366    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-token\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.027775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.027417    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-envoy\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.133488    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-credential-socket\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.133602    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f416feb-13be-4a2c-b2bc-9999484d1d56-istiod-ca-cert\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.133709    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-certs\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.133734    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-podinfo\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.133876    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-data\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.133931    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-socket\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.134081    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vz8n\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-kube-api-access-7vz8n\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.134143    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-token\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.135744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.134190    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-envoy\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.136359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.136260    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f416feb-13be-4a2c-b2bc-9999484d1d56-istiod-ca-cert\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.136546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.136519    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-credential-socket\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.136826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.136801    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-certs\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.162728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.137908    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-socket\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.162728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.138123    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-data\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.162728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.161001    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-envoy\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.174691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.169254    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-token\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.174691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.169779    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-podinfo\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.199738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.198523    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vz8n\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-kube-api-access-7vz8n\") pod \"e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") " pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.307494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.307366    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:08.441311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.441270    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:20:08.544099 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544066    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-certs\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544111    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-podinfo\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544157    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istiod-ca-cert\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544190    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-token\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-token\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544302    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-credential-socket\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544355    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-socket\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544380    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-certs" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:08.544574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544418    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-envoy\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544572    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-data\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544610    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-kube-api-access-628wp\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-kube-api-access-628wp\") pod \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\" (UID: \"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f\") "
Oct 30 21:20:08.544769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544630    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-socket" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:08.544769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544694    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istiod-ca-cert" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:08.544942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544831    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-data" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:08.544942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.544896    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-credential-socket" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:08.545622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.545596    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.545779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.545625    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.545779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.545641    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.545779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.545656    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.545779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.545690    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.546582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.546558    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-podinfo" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:20:08.546755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.546609    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-token" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:08.546831 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.546805    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-kube-api-access-628wp" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "kube-api-access-628wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:08.546910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.546894    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-envoy" pod "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" (UID: "3f008a7d-ba68-4238-9dee-3bc3d6b56a9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:08.646793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.646766    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-628wp\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-kube-api-access-628wp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.646793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.646789    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.646793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.646802    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.647022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.646810    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:08.653401 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:08.653363    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f416feb_13be_4a2c_b2bc_9999484d1d56.slice/crio-109ed81ab0fcf4431a9403e8fc10d02bf4dcd05505ab142575e2475447c99d40 WatchSource:0}: Error finding container 109ed81ab0fcf4431a9403e8fc10d02bf4dcd05505ab142575e2475447c99d40: Status 404 returned error can't find the container with id 109ed81ab0fcf4431a9403e8fc10d02bf4dcd05505ab142575e2475447c99d40
Oct 30 21:20:08.653401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.653380    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"]
Oct 30 21:20:08.655975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.655948    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:08.656036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.656007    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:08.656075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.656037    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:08.689311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.689281    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" event={"ID":"4f416feb-13be-4a2c-b2bc-9999484d1d56","Type":"ContainerStarted","Data":"109ed81ab0fcf4431a9403e8fc10d02bf4dcd05505ab142575e2475447c99d40"}
Oct 30 21:20:08.691384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.691352    4633 generic.go:356] "Generic (PLEG): container finished" podID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerID="a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90" exitCode=0
Oct 30 21:20:08.691500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.691398    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" event={"ID":"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f","Type":"ContainerDied","Data":"a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90"}
Oct 30 21:20:08.691500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.691406    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"
Oct 30 21:20:08.691500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.691445    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28" event={"ID":"3f008a7d-ba68-4238-9dee-3bc3d6b56a9f","Type":"ContainerDied","Data":"0ec42d30da01fa8e38f1d8d97ee4ee156fdb3d84dc42c94c7fb053cb95406a35"}
Oct 30 21:20:08.691500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.691476    4633 scope.go:122] "RemoveContainer" containerID="a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90"
Oct 30 21:20:08.713508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.713480    4633 scope.go:122] "RemoveContainer" containerID="405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff"
Oct 30 21:20:08.725550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.725521    4633 scope.go:122] "RemoveContainer" containerID="a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90"
Oct 30 21:20:08.726548 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:08.726464    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90\": container with ID starting with a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90 not found: ID does not exist" containerID="a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90"
Oct 30 21:20:08.726682 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.726574    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90"} err="failed to get container status \"a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90\": rpc error: code = NotFound desc = could not find container \"a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90\": container with ID starting with a51c4373cca7f116467353cae3aa2eb226acbc8eb702dc3267aac60ddc657e90 not found: ID does not exist"
Oct 30 21:20:08.726682 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.726607    4633 scope.go:122] "RemoveContainer" containerID="405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff"
Oct 30 21:20:08.727027 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:08.726992    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff\": container with ID starting with 405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff not found: ID does not exist" containerID="405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff"
Oct 30 21:20:08.727113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.727036    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff"} err="failed to get container status \"405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff\": rpc error: code = NotFound desc = could not find container \"405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff\": container with ID starting with 405434688fa7705854154f66159f0994d513d87d2cb1911531a446f599907dff not found: ID does not exist"
Oct 30 21:20:08.727271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.727251    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"]
Oct 30 21:20:08.731500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:08.731480    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0c4aac-openshift-default-7cc5657677-4nt28"]
Oct 30 21:20:09.701556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.701519    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" event={"ID":"4f416feb-13be-4a2c-b2bc-9999484d1d56","Type":"ContainerStarted","Data":"534b1fa378bfe125e4d4b529ead2b1c62a51ce3ed36771f18b89393062f1dfb5"}
Oct 30 21:20:09.792629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.792581    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" path="/var/lib/kubelet/pods/3f008a7d-ba68-4238-9dee-3bc3d6b56a9f/volumes"
Oct 30 21:20:09.898702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.898504    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podStartSLOduration=2.898482196 podStartE2EDuration="2.898482196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:09.72369737 +0000 UTC m=+1118.073424757" watchObservedRunningTime="2026-10-30 21:20:09.898482196 +0000 UTC m=+1118.248209593"
Oct 30 21:20:09.905775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.902902    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"]
Oct 30 21:20:09.905775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.904168    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy"
Oct 30 21:20:09.905775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.904189    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy"
Oct 30 21:20:09.905775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.904579    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy"
Oct 30 21:20:09.923360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.923232    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.929188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.928408    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3ac6bf-openshift-default-dockercfg-vznzc\""
Oct 30 21:20:09.940477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.930622    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"]
Oct 30 21:20:09.940477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.931777    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy"
Oct 30 21:20:09.940477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.931809    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy"
Oct 30 21:20:09.940477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.932015    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f008a7d-ba68-4238-9dee-3bc3d6b56a9f" containerName="istio-proxy"
Oct 30 21:20:09.940850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.940729    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:09.945722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.945555    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"]
Oct 30 21:20:09.951783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.951297    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"]
Oct 30 21:20:09.988709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.988653    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.988883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.988727    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.988883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.988759    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nlfb\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-kube-api-access-9nlfb\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.988883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.988788    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.988883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.988811    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.989088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.989011    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.989088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.989061    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.989213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.989098    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:09.989269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:09.989222    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090172    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090201    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090225    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nlfb\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-kube-api-access-9nlfb\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090243    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090265    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090297    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.090424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090380    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdd7g\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-kube-api-access-hdd7g\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.090750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090440    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.090750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090566    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090569    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.090750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090610    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090639    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.090750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090694    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec77c66-fcd5-43c0-aa30-55d522a0a157-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.090750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090746    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.091129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090832    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.091129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090879    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.091129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.090883    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.091129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.091037    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.091129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.091070    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.091129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.091105    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.091458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.091289    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.091458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.091379    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.091458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.091431    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.092901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.092880    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.093390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.093371    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.097685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.097654    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.097843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.097825    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nlfb\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-kube-api-access-9nlfb\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.196015 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.195968    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdd7g\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-kube-api-access-hdd7g\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.196217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.196029    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.196217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.196073    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.196217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.196107    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec77c66-fcd5-43c0-aa30-55d522a0a157-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.196217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.196175    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.196434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.196303    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.196434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.196340    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.196541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.196461    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.196541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.196524    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.198157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.198123    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.198453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.198418    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.199553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.199529    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.199758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.199629    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.202630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.202505    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.202950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.202926    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.204014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.203988    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec77c66-fcd5-43c0-aa30-55d522a0a157-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.207425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.206170    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.208088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.208022    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdd7g\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-kube-api-access-hdd7g\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.259479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.259447    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:10.268264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.268243    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:10.309810 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.309773    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:10.318635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.318407    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:10.564002 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:10.563974    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc53139_a140_4d72_a29f_fda75ddb7d5e.slice/crio-ad6b61902f940578aa3a762050c4f862222926585781aa08bd723f02b898f05d WatchSource:0}: Error finding container ad6b61902f940578aa3a762050c4f862222926585781aa08bd723f02b898f05d: Status 404 returned error can't find the container with id ad6b61902f940578aa3a762050c4f862222926585781aa08bd723f02b898f05d
Oct 30 21:20:10.564164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.564147    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"]
Oct 30 21:20:10.564987 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:10.564925    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec77c66_fcd5_43c0_aa30_55d522a0a157.slice/crio-6982e6ac1b76ea27eec58d9e4b8214a2e2ed051942946ba3d0db7864f6f6456d WatchSource:0}: Error finding container 6982e6ac1b76ea27eec58d9e4b8214a2e2ed051942946ba3d0db7864f6f6456d: Status 404 returned error can't find the container with id 6982e6ac1b76ea27eec58d9e4b8214a2e2ed051942946ba3d0db7864f6f6456d
Oct 30 21:20:10.567130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.567095    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:10.567214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.567176    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:10.567254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.567218    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:10.567716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.567648    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:10.567832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.567767    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:10.567832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.567808    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:10.569194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.569174    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"]
Oct 30 21:20:10.714497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.714448    4633 generic.go:356] "Generic (PLEG): container finished" podID="95881bbb-04ab-44f1-a15b-f0c90ca72d84" containerID="06bcaacbf84a25eb5adc3e97e017e47591d97c0d11e6168f589b2801857e5cdd" exitCode=0
Oct 30 21:20:10.715013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.714495    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx" event={"ID":"95881bbb-04ab-44f1-a15b-f0c90ca72d84","Type":"ContainerDied","Data":"06bcaacbf84a25eb5adc3e97e017e47591d97c0d11e6168f589b2801857e5cdd"}
Oct 30 21:20:10.717139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.717108    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" event={"ID":"8fc53139-a140-4d72-a29f-fda75ddb7d5e","Type":"ContainerStarted","Data":"f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7"}
Oct 30 21:20:10.717293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.717164    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" event={"ID":"8fc53139-a140-4d72-a29f-fda75ddb7d5e","Type":"ContainerStarted","Data":"ad6b61902f940578aa3a762050c4f862222926585781aa08bd723f02b898f05d"}
Oct 30 21:20:10.721763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.721716    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" event={"ID":"bec77c66-fcd5-43c0-aa30-55d522a0a157","Type":"ContainerStarted","Data":"8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029"}
Oct 30 21:20:10.721953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.721790    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:10.722017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.721982    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" event={"ID":"bec77c66-fcd5-43c0-aa30-55d522a0a157","Type":"ContainerStarted","Data":"6982e6ac1b76ea27eec58d9e4b8214a2e2ed051942946ba3d0db7864f6f6456d"}
Oct 30 21:20:10.722423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.722407    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:10.752488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.752437    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podStartSLOduration=1.752421509 podStartE2EDuration="1.752421509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:10.748905447 +0000 UTC m=+1119.098632845" watchObservedRunningTime="2026-10-30 21:20:10.752421509 +0000 UTC m=+1119.102148905"
Oct 30 21:20:10.804956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:10.804699    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podStartSLOduration=1.804658566 podStartE2EDuration="1.804658566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:10.798714447 +0000 UTC m=+1119.148441848" watchObservedRunningTime="2026-10-30 21:20:10.804658566 +0000 UTC m=+1119.154385963"
Oct 30 21:20:11.264393 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:11.264363    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:11.264781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:11.264759    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:20:11.264954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:11.264931    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:20:11.279599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:11.278923    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:11.279990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:11.279967    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:20:11.280159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:11.280136    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:20:12.261910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.261822    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:20:12.261910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.261889    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:20:12.269720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.269691    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:20:12.269855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.269747    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:20:12.295607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.295582    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:20:12.361268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.361164    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95881bbb-04ab-44f1-a15b-f0c90ca72d84-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95881bbb-04ab-44f1-a15b-f0c90ca72d84-archives-path\") pod \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") "
Oct 30 21:20:12.361439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.361298    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-service-ca-bundle\") pod \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") "
Oct 30 21:20:12.361499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.361438    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95881bbb-04ab-44f1-a15b-f0c90ca72d84-kube-api-access-k7tdl\" (UniqueName: \"kubernetes.io/projected/95881bbb-04ab-44f1-a15b-f0c90ca72d84-kube-api-access-k7tdl\") pod \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") "
Oct 30 21:20:12.361562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.361510    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-trusted-ca-bundle\") pod \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\" (UID: \"95881bbb-04ab-44f1-a15b-f0c90ca72d84\") "
Oct 30 21:20:12.362215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.362189    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95881bbb-04ab-44f1-a15b-f0c90ca72d84-archives-path" pod "95881bbb-04ab-44f1-a15b-f0c90ca72d84" (UID: "95881bbb-04ab-44f1-a15b-f0c90ca72d84"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:12.362556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.362528    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-trusted-ca-bundle" pod "95881bbb-04ab-44f1-a15b-f0c90ca72d84" (UID: "95881bbb-04ab-44f1-a15b-f0c90ca72d84"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:12.362999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.362976    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-service-ca-bundle" pod "95881bbb-04ab-44f1-a15b-f0c90ca72d84" (UID: "95881bbb-04ab-44f1-a15b-f0c90ca72d84"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:12.370318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.370271    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95881bbb-04ab-44f1-a15b-f0c90ca72d84-kube-api-access-k7tdl" pod "95881bbb-04ab-44f1-a15b-f0c90ca72d84" (UID: "95881bbb-04ab-44f1-a15b-f0c90ca72d84"). InnerVolumeSpecName "kube-api-access-k7tdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:12.466108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.466061    4633 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/95881bbb-04ab-44f1-a15b-f0c90ca72d84-archives-path\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:12.466108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.466104    4633 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-service-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:12.466381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.466121    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7tdl\" (UniqueName: \"kubernetes.io/projected/95881bbb-04ab-44f1-a15b-f0c90ca72d84-kube-api-access-k7tdl\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:12.466381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.466139    4633 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95881bbb-04ab-44f1-a15b-f0c90ca72d84-trusted-ca-bundle\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:12.788677 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.788650    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx"
Oct 30 21:20:12.788858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.788694    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vbsd2-tn5zx" event={"ID":"95881bbb-04ab-44f1-a15b-f0c90ca72d84","Type":"ContainerDied","Data":"9d1ca9e61e00c4e0b6df86fdbd9a31b14c1d30649a592688629af5f45f6d70f6"}
Oct 30 21:20:12.788858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:12.788736    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d1ca9e61e00c4e0b6df86fdbd9a31b14c1d30649a592688629af5f45f6d70f6"
Oct 30 21:20:13.263701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.263148    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:20:13.263701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.263209    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:20:13.268633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.268601    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:20:13.268780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.268659    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:20:13.669871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.669841    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"]
Oct 30 21:20:13.670781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.670757    4633 cpu_manager.go:414] "containerMap: removing container" podUID="95881bbb-04ab-44f1-a15b-f0c90ca72d84" containerName="insights-gathering"
Oct 30 21:20:13.670781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.670780    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95881bbb-04ab-44f1-a15b-f0c90ca72d84" containerName="insights-gathering"
Oct 30 21:20:13.670991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.670954    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="95881bbb-04ab-44f1-a15b-f0c90ca72d84" containerName="insights-gathering"
Oct 30 21:20:13.675318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.675287    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:13.676958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.676867    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"]
Oct 30 21:20:13.682736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.682716    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"]
Oct 30 21:20:13.682937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.682864    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:13.687238 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.687214    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"]
Oct 30 21:20:13.689846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.689822    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-43f6ab-dockercfg-lhhzj\""
Oct 30 21:20:13.690754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.690732    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-43f6ab\""
Oct 30 21:20:13.790332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.790299    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccde7d8-1dae-40bd-bd59-a3668da8f24d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg\" (UID: \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:13.790721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.790692    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb47f\" (UniqueName: \"kubernetes.io/projected/b9439f93-cd0f-4b03-b234-e354e15436a7-kube-api-access-jb47f\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t\" (UID: \"b9439f93-cd0f-4b03-b234-e354e15436a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:13.791575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.791554    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9439f93-cd0f-4b03-b234-e354e15436a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t\" (UID: \"b9439f93-cd0f-4b03-b234-e354e15436a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:13.805858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.805820    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcs7\" (UniqueName: \"kubernetes.io/projected/cccde7d8-1dae-40bd-bd59-a3668da8f24d-kube-api-access-fhcs7\") pod \"payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg\" (UID: \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:13.932097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.932003    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb47f\" (UniqueName: \"kubernetes.io/projected/b9439f93-cd0f-4b03-b234-e354e15436a7-kube-api-access-jb47f\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t\" (UID: \"b9439f93-cd0f-4b03-b234-e354e15436a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:13.962788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.932357    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9439f93-cd0f-4b03-b234-e354e15436a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t\" (UID: \"b9439f93-cd0f-4b03-b234-e354e15436a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:13.962788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.937772    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcs7\" (UniqueName: \"kubernetes.io/projected/cccde7d8-1dae-40bd-bd59-a3668da8f24d-kube-api-access-fhcs7\") pod \"payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg\" (UID: \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:13.962788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.938116    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccde7d8-1dae-40bd-bd59-a3668da8f24d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg\" (UID: \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:13.962788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.939205    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccde7d8-1dae-40bd-bd59-a3668da8f24d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg\" (UID: \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:13.962788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:13.935738    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9439f93-cd0f-4b03-b234-e354e15436a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t\" (UID: \"b9439f93-cd0f-4b03-b234-e354e15436a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:14.027571 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.027057    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcs7\" (UniqueName: \"kubernetes.io/projected/cccde7d8-1dae-40bd-bd59-a3668da8f24d-kube-api-access-fhcs7\") pod \"payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg\" (UID: \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:14.078855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.078817    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb47f\" (UniqueName: \"kubernetes.io/projected/b9439f93-cd0f-4b03-b234-e354e15436a7-kube-api-access-jb47f\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t\" (UID: \"b9439f93-cd0f-4b03-b234-e354e15436a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:14.263187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.263091    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:20:14.263446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.263415    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:20:14.269602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.269565    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:20:14.269941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.269621    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:20:14.325750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.311956    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:14.339388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.329089    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:14.693572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.693538    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"]
Oct 30 21:20:14.711858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:14.711823    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"]
Oct 30 21:20:15.071827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:15.070763    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"]
Oct 30 21:20:15.077285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:15.077260    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"]
Oct 30 21:20:15.085089 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:15.085059    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccde7d8_1dae_40bd_bd59_a3668da8f24d.slice/crio-ca37ef0c6c8b35d0ed0a207e0d28e1102e2d769b715e8b6d2edecf64bdaece44 WatchSource:0}: Error finding container ca37ef0c6c8b35d0ed0a207e0d28e1102e2d769b715e8b6d2edecf64bdaece44: Status 404 returned error can't find the container with id ca37ef0c6c8b35d0ed0a207e0d28e1102e2d769b715e8b6d2edecf64bdaece44
Oct 30 21:20:15.260694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:15.260103    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:20:15.260694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:15.260238    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:20:15.271737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:15.271076    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:20:15.271737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:15.271136    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:20:16.107147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.107112    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg" event={"ID":"cccde7d8-1dae-40bd-bd59-a3668da8f24d","Type":"ContainerStarted","Data":"efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65"}
Oct 30 21:20:16.107361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.107174    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg" event={"ID":"cccde7d8-1dae-40bd-bd59-a3668da8f24d","Type":"ContainerStarted","Data":"ca37ef0c6c8b35d0ed0a207e0d28e1102e2d769b715e8b6d2edecf64bdaece44"}
Oct 30 21:20:16.107361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.107248    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:16.110207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.110176    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t" event={"ID":"b9439f93-cd0f-4b03-b234-e354e15436a7","Type":"ContainerStarted","Data":"a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c"}
Oct 30 21:20:16.110330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.110230    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t" event={"ID":"b9439f93-cd0f-4b03-b234-e354e15436a7","Type":"ContainerStarted","Data":"62159a013f86ac7d5fc71a0c30bfc320726b2d01f7c96decc628cb7931e14438"}
Oct 30 21:20:16.110368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.110350    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:16.166694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.164496    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg" podStartSLOduration=3.16447188 podStartE2EDuration="3.16447188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:16.125070606 +0000 UTC m=+1124.474798005" watchObservedRunningTime="2026-10-30 21:20:16.16447188 +0000 UTC m=+1124.514199277"
Oct 30 21:20:16.166694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.166376    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t" podStartSLOduration=3.166362082 podStartE2EDuration="3.166362082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:16.16515008 +0000 UTC m=+1124.514877479" watchObservedRunningTime="2026-10-30 21:20:16.166362082 +0000 UTC m=+1124.516089478"
Oct 30 21:20:16.269697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.269052    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:20:16.269697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.269116    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:20:16.270418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.270087    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:20:16.270418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.270132    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:20:16.382867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.382764    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"]
Oct 30 21:20:16.389755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.388540    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"]
Oct 30 21:20:16.389755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.388690    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:16.393691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.393641    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60c26c\""
Oct 30 21:20:16.393824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.393712    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"]
Oct 30 21:20:16.398241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.398208    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb\" (UID: \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:16.398369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.398352    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x54kn\" (UniqueName: \"kubernetes.io/projected/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-kube-api-access-x54kn\") pod \"payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb\" (UID: \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:16.399246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.399223    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"]
Oct 30 21:20:16.399354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.399335    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:16.406617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.405502    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60c26c-dockercfg-g95sl\""
Oct 30 21:20:16.544700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.513327    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb\" (UID: \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:16.544700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.513453    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x54kn\" (UniqueName: \"kubernetes.io/projected/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-kube-api-access-x54kn\") pod \"payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb\" (UID: \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:16.554719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.545272    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb\" (UID: \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:16.630695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.629459    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x54kn\" (UniqueName: \"kubernetes.io/projected/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-kube-api-access-x54kn\") pod \"payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb\" (UID: \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:16.732593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.731236    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:16.732593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.731965    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd40b3f-da73-4089-a78c-8699dd3e5077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s\" (UID: \"afd40b3f-da73-4089-a78c-8699dd3e5077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:16.732593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.732099    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6566\" (UniqueName: \"kubernetes.io/projected/afd40b3f-da73-4089-a78c-8699dd3e5077-kube-api-access-f6566\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s\" (UID: \"afd40b3f-da73-4089-a78c-8699dd3e5077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:16.907851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.907811    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd40b3f-da73-4089-a78c-8699dd3e5077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s\" (UID: \"afd40b3f-da73-4089-a78c-8699dd3e5077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:16.908453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.908424    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6566\" (UniqueName: \"kubernetes.io/projected/afd40b3f-da73-4089-a78c-8699dd3e5077-kube-api-access-f6566\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s\" (UID: \"afd40b3f-da73-4089-a78c-8699dd3e5077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:16.908730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.908709    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd40b3f-da73-4089-a78c-8699dd3e5077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s\" (UID: \"afd40b3f-da73-4089-a78c-8699dd3e5077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:16.940765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:16.940733    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6566\" (UniqueName: \"kubernetes.io/projected/afd40b3f-da73-4089-a78c-8699dd3e5077-kube-api-access-f6566\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s\" (UID: \"afd40b3f-da73-4089-a78c-8699dd3e5077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:17.210212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:17.210186    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:17.277411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:17.277368    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 30 21:20:17.277569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:17.277432    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 30 21:20:17.278210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:17.278177    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 21:20:17.278340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:17.278241    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 21:20:17.285316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:17.285236    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"]
Oct 30 21:20:17.634518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:17.634487    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"]
Oct 30 21:20:17.636381 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:17.636355    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd40b3f_da73_4089_a78c_8699dd3e5077.slice/crio-14c9ea9c17fc49a3fde3368365591824c45ed22f57236aae6afd3d532745a92b WatchSource:0}: Error finding container 14c9ea9c17fc49a3fde3368365591824c45ed22f57236aae6afd3d532745a92b: Status 404 returned error can't find the container with id 14c9ea9c17fc49a3fde3368365591824c45ed22f57236aae6afd3d532745a92b
Oct 30 21:20:18.264847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.264816    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:18.265235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.265216    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:18.265916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.265896    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:20:18.273460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.273440    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:18.273723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.273704    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:18.274590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.274573    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:20:18.288097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.288062    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb" event={"ID":"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9","Type":"ContainerStarted","Data":"d47d91b98c67eda67b58a744930ab59720a25283cc189341a26e53fb2f5fc9ac"}
Oct 30 21:20:18.288278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.288118    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb" event={"ID":"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9","Type":"ContainerStarted","Data":"556889f7f31d2f63b7f937d37f377311abcae39b1f1041019001c998524f0972"}
Oct 30 21:20:18.288278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.288261    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:18.289922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.289898    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s" event={"ID":"afd40b3f-da73-4089-a78c-8699dd3e5077","Type":"ContainerStarted","Data":"da51b7e5f6b0511b77637548d5bf4d2960b8a3cea7dc63b62f50a514e724d6b7"}
Oct 30 21:20:18.290027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.289947    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s" event={"ID":"afd40b3f-da73-4089-a78c-8699dd3e5077","Type":"ContainerStarted","Data":"14c9ea9c17fc49a3fde3368365591824c45ed22f57236aae6afd3d532745a92b"}
Oct 30 21:20:18.290147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.290132    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:18.348068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.348011    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb" podStartSLOduration=2.347991674 podStartE2EDuration="2.347991674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:18.339874176 +0000 UTC m=+1126.689601574" watchObservedRunningTime="2026-10-30 21:20:18.347991674 +0000 UTC m=+1126.697719072"
Oct 30 21:20:18.361422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:18.361359    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s" podStartSLOduration=2.36134193 podStartE2EDuration="2.36134193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:18.356338924 +0000 UTC m=+1126.706066337" watchObservedRunningTime="2026-10-30 21:20:18.36134193 +0000 UTC m=+1126.711069330"
Oct 30 21:20:19.694353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:19.694300    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8" podUID="42781959-fe94-4542-aa6c-57f3b7a3e180" containerName="payload-processing" containerID="cri-o://4f2d6e964ab185cf0949d39427b6563820cb2845b5730c5cfb7fe68e3f09ea25" gracePeriod=25
Oct 30 21:20:19.712515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:19.712486    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86" podUID="281b139e-c545-4213-ab8b-b83fafff473e" containerName="payload-pre-processing" containerID="cri-o://be113fba5d0c456f2dd88f41e1acc5940ae6dcb9cd225eac6808f8ad4bb4b585" gracePeriod=25
Oct 30 21:20:19.889313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:19.889279    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:20:20.389828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.389798    4633 generic.go:356] "Generic (PLEG): container finished" podID="42781959-fe94-4542-aa6c-57f3b7a3e180" containerID="4f2d6e964ab185cf0949d39427b6563820cb2845b5730c5cfb7fe68e3f09ea25" exitCode=0
Oct 30 21:20:20.390192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.390093    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8" event={"ID":"42781959-fe94-4542-aa6c-57f3b7a3e180","Type":"ContainerDied","Data":"4f2d6e964ab185cf0949d39427b6563820cb2845b5730c5cfb7fe68e3f09ea25"}
Oct 30 21:20:20.410809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.410768    4633 generic.go:356] "Generic (PLEG): container finished" podID="281b139e-c545-4213-ab8b-b83fafff473e" containerID="be113fba5d0c456f2dd88f41e1acc5940ae6dcb9cd225eac6808f8ad4bb4b585" exitCode=0
Oct 30 21:20:20.410951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.410871    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86" event={"ID":"281b139e-c545-4213-ab8b-b83fafff473e","Type":"ContainerDied","Data":"be113fba5d0c456f2dd88f41e1acc5940ae6dcb9cd225eac6808f8ad4bb4b585"}
Oct 30 21:20:20.422559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.422524    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:20:20.432220 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.430161    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:20:20.477905 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.477871    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281b139e-c545-4213-ab8b-b83fafff473e-kube-api-access-pm55j\" (UniqueName: \"kubernetes.io/projected/281b139e-c545-4213-ab8b-b83fafff473e-kube-api-access-pm55j\") pod \"281b139e-c545-4213-ab8b-b83fafff473e\" (UID: \"281b139e-c545-4213-ab8b-b83fafff473e\") "
Oct 30 21:20:20.478079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.478017    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/281b139e-c545-4213-ab8b-b83fafff473e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281b139e-c545-4213-ab8b-b83fafff473e-plugins-config-volume\") pod \"281b139e-c545-4213-ab8b-b83fafff473e\" (UID: \"281b139e-c545-4213-ab8b-b83fafff473e\") "
Oct 30 21:20:20.478166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.478146    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42781959-fe94-4542-aa6c-57f3b7a3e180-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42781959-fe94-4542-aa6c-57f3b7a3e180-plugins-config-volume\") pod \"42781959-fe94-4542-aa6c-57f3b7a3e180\" (UID: \"42781959-fe94-4542-aa6c-57f3b7a3e180\") "
Oct 30 21:20:20.478279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.478257    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42781959-fe94-4542-aa6c-57f3b7a3e180-kube-api-access-wm26v\" (UniqueName: \"kubernetes.io/projected/42781959-fe94-4542-aa6c-57f3b7a3e180-kube-api-access-wm26v\") pod \"42781959-fe94-4542-aa6c-57f3b7a3e180\" (UID: \"42781959-fe94-4542-aa6c-57f3b7a3e180\") "
Oct 30 21:20:20.478644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.478607    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281b139e-c545-4213-ab8b-b83fafff473e-plugins-config-volume" pod "281b139e-c545-4213-ab8b-b83fafff473e" (UID: "281b139e-c545-4213-ab8b-b83fafff473e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:20.478644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.478631    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42781959-fe94-4542-aa6c-57f3b7a3e180-plugins-config-volume" pod "42781959-fe94-4542-aa6c-57f3b7a3e180" (UID: "42781959-fe94-4542-aa6c-57f3b7a3e180"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:20.479150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.479124    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281b139e-c545-4213-ab8b-b83fafff473e-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:20.479224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.479156    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42781959-fe94-4542-aa6c-57f3b7a3e180-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:20.481000 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.480977    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42781959-fe94-4542-aa6c-57f3b7a3e180-kube-api-access-wm26v" pod "42781959-fe94-4542-aa6c-57f3b7a3e180" (UID: "42781959-fe94-4542-aa6c-57f3b7a3e180"). InnerVolumeSpecName "kube-api-access-wm26v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:20.481106 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.481009    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281b139e-c545-4213-ab8b-b83fafff473e-kube-api-access-pm55j" pod "281b139e-c545-4213-ab8b-b83fafff473e" (UID: "281b139e-c545-4213-ab8b-b83fafff473e"). InnerVolumeSpecName "kube-api-access-pm55j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:20.580618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.580578    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm55j\" (UniqueName: \"kubernetes.io/projected/281b139e-c545-4213-ab8b-b83fafff473e-kube-api-access-pm55j\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:20.580618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:20.580620    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm26v\" (UniqueName: \"kubernetes.io/projected/42781959-fe94-4542-aa6c-57f3b7a3e180-kube-api-access-wm26v\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:21.251373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.251325    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8" podUID="42781959-fe94-4542-aa6c-57f3b7a3e180" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.119:50052\" within 1s: context deadline exceeded"
Oct 30 21:20:21.252567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.252054    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86" podUID="281b139e-c545-4213-ab8b-b83fafff473e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.120:50052\" within 1s: context deadline exceeded"
Oct 30 21:20:21.252808 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:21.252073    4633 logging.go:55] [core] [Channel #1360 SubChannel #1361]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.119:50052", ServerName: "10.133.0.119:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.119:50052: operation was canceled"
Oct 30 21:20:21.252884 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:21.252147    4633 logging.go:55] [core] [Channel #1358 SubChannel #1359]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.120:50052", ServerName: "10.133.0.120:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.120:50052: operation was canceled"
Oct 30 21:20:21.422623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.422564    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8" event={"ID":"42781959-fe94-4542-aa6c-57f3b7a3e180","Type":"ContainerDied","Data":"c79a34808c83ea3cc6f1016f9bd5cc7b1520cdb1f0991ae0bdd561383cc78204"}
Oct 30 21:20:21.422854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.422636    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"
Oct 30 21:20:21.422854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.422717    4633 scope.go:122] "RemoveContainer" containerID="4f2d6e964ab185cf0949d39427b6563820cb2845b5730c5cfb7fe68e3f09ea25"
Oct 30 21:20:21.426053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.425436    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86" event={"ID":"281b139e-c545-4213-ab8b-b83fafff473e","Type":"ContainerDied","Data":"d7fcb7a405e6f08a377e78027a82c2f28534082117d913cc4163d1ac14581c8f"}
Oct 30 21:20:21.426053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.425554    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"
Oct 30 21:20:21.438739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.438718    4633 scope.go:122] "RemoveContainer" containerID="be113fba5d0c456f2dd88f41e1acc5940ae6dcb9cd225eac6808f8ad4bb4b585"
Oct 30 21:20:21.463898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.462765    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"]
Oct 30 21:20:21.469782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.469754    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e494a9-8854f8cbf-vwzq8"]
Oct 30 21:20:21.474203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.474180    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"]
Oct 30 21:20:21.478573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.478546    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e494a9-56445b494c-97g86"]
Oct 30 21:20:21.811387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.811350    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281b139e-c545-4213-ab8b-b83fafff473e" path="/var/lib/kubelet/pods/281b139e-c545-4213-ab8b-b83fafff473e/volumes"
Oct 30 21:20:21.812109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:21.812083    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42781959-fe94-4542-aa6c-57f3b7a3e180" path="/var/lib/kubelet/pods/42781959-fe94-4542-aa6c-57f3b7a3e180/volumes"
Oct 30 21:20:23.075832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.075786    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"]
Oct 30 21:20:23.077026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.077004    4633 cpu_manager.go:414] "containerMap: removing container" podUID="42781959-fe94-4542-aa6c-57f3b7a3e180" containerName="payload-processing"
Oct 30 21:20:23.077108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.077032    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42781959-fe94-4542-aa6c-57f3b7a3e180" containerName="payload-processing"
Oct 30 21:20:23.077108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.077056    4633 cpu_manager.go:414] "containerMap: removing container" podUID="281b139e-c545-4213-ab8b-b83fafff473e" containerName="payload-pre-processing"
Oct 30 21:20:23.077108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.077064    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281b139e-c545-4213-ab8b-b83fafff473e" containerName="payload-pre-processing"
Oct 30 21:20:23.077257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.077232    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="42781959-fe94-4542-aa6c-57f3b7a3e180" containerName="payload-processing"
Oct 30 21:20:23.077257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.077248    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="281b139e-c545-4213-ab8b-b83fafff473e" containerName="payload-pre-processing"
Oct 30 21:20:23.086916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.085357    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.099140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.099109    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"]
Oct 30 21:20:23.108227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.108185    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-60c26c-serving-cert\""
Oct 30 21:20:23.175982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.175947    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec016bee-4fb5-4681-9ec6-7b9633d42af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.175982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.175986    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4r6\" (UniqueName: \"kubernetes.io/projected/ec016bee-4fb5-4681-9ec6-7b9633d42af6-kube-api-access-2m4r6\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.176199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.176145    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.176283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.176263    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-metrics-certs\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.281585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.280287    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec016bee-4fb5-4681-9ec6-7b9633d42af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.281585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.280342    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4r6\" (UniqueName: \"kubernetes.io/projected/ec016bee-4fb5-4681-9ec6-7b9633d42af6-kube-api-access-2m4r6\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.281585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.280449    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.281585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.280570    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-metrics-certs\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.281952 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:23.281848    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-serving-cert: secret "maas-api-e2e-ipp-b-60c26c-serving-cert" not found
Oct 30 21:20:23.281952 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:23.281927    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls podName:ec016bee-4fb5-4681-9ec6-7b9633d42af6 nodeName:}" failed. No retries permitted until 2026-10-30 21:20:23.781902896 +0000 UTC m=+1132.131630283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls") pod "maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn" (UID: "ec016bee-4fb5-4681-9ec6-7b9633d42af6") : secret "maas-api-e2e-ipp-b-60c26c-serving-cert" not found
Oct 30 21:20:23.282936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.282905    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec016bee-4fb5-4681-9ec6-7b9633d42af6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.310695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.305786    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-metrics-certs\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.316939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.316894    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4r6\" (UniqueName: \"kubernetes.io/projected/ec016bee-4fb5-4681-9ec6-7b9633d42af6-kube-api-access-2m4r6\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.795690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.792748    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:23.812343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:23.812305    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:24.019525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:24.019468    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:24.141803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:24.140172    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"]
Oct 30 21:20:24.141803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:24.140419    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5" podUID="348cbb0a-d675-4025-bc1e-b13afd70dd5c" containerName="maas-api" containerID="cri-o://72cb2f71eb94e82620e7f552beedaba0f512383b0dae6d45f185a378abcdbe5a" gracePeriod=30
Oct 30 21:20:24.595283 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:24.588970    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec016bee_4fb5_4681_9ec6_7b9633d42af6.slice/crio-efa8bfa4d9937552dfc33837775bebc70fe5bd9f8744532500d08980bdd5f924 WatchSource:0}: Error finding container efa8bfa4d9937552dfc33837775bebc70fe5bd9f8744532500d08980bdd5f924: Status 404 returned error can't find the container with id efa8bfa4d9937552dfc33837775bebc70fe5bd9f8744532500d08980bdd5f924
Oct 30 21:20:24.680472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:24.680437    4633 generic.go:356] "Generic (PLEG): container finished" podID="348cbb0a-d675-4025-bc1e-b13afd70dd5c" containerID="72cb2f71eb94e82620e7f552beedaba0f512383b0dae6d45f185a378abcdbe5a" exitCode=0
Oct 30 21:20:24.691845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:24.680743    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5" event={"ID":"348cbb0a-d675-4025-bc1e-b13afd70dd5c","Type":"ContainerDied","Data":"72cb2f71eb94e82620e7f552beedaba0f512383b0dae6d45f185a378abcdbe5a"}
Oct 30 21:20:24.817691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:24.817633    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"]
Oct 30 21:20:25.077390 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:25.077354    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:20:25.110851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.110822    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:20:25.263439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.263409    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348cbb0a-d675-4025-bc1e-b13afd70dd5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348cbb0a-d675-4025-bc1e-b13afd70dd5c-openshift-service-ca\") pod \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") "
Oct 30 21:20:25.263927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.263471    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-maas-api-tls\") pod \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") "
Oct 30 21:20:25.263927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.263540    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-metrics-certs\") pod \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") "
Oct 30 21:20:25.263927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.263826    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348cbb0a-d675-4025-bc1e-b13afd70dd5c-kube-api-access-grmr5\" (UniqueName: \"kubernetes.io/projected/348cbb0a-d675-4025-bc1e-b13afd70dd5c-kube-api-access-grmr5\") pod \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\" (UID: \"348cbb0a-d675-4025-bc1e-b13afd70dd5c\") "
Oct 30 21:20:25.263927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.263879    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348cbb0a-d675-4025-bc1e-b13afd70dd5c-openshift-service-ca" pod "348cbb0a-d675-4025-bc1e-b13afd70dd5c" (UID: "348cbb0a-d675-4025-bc1e-b13afd70dd5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:25.264534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.264510    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348cbb0a-d675-4025-bc1e-b13afd70dd5c-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:25.267283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.267258    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-metrics-certs" pod "348cbb0a-d675-4025-bc1e-b13afd70dd5c" (UID: "348cbb0a-d675-4025-bc1e-b13afd70dd5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:25.267439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.267412    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-maas-api-tls" pod "348cbb0a-d675-4025-bc1e-b13afd70dd5c" (UID: "348cbb0a-d675-4025-bc1e-b13afd70dd5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:25.267714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.267651    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348cbb0a-d675-4025-bc1e-b13afd70dd5c-kube-api-access-grmr5" pod "348cbb0a-d675-4025-bc1e-b13afd70dd5c" (UID: "348cbb0a-d675-4025-bc1e-b13afd70dd5c"). InnerVolumeSpecName "kube-api-access-grmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:25.366392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.366309    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grmr5\" (UniqueName: \"kubernetes.io/projected/348cbb0a-d675-4025-bc1e-b13afd70dd5c-kube-api-access-grmr5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:25.366392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.366343    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:25.366392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.366352    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348cbb0a-d675-4025-bc1e-b13afd70dd5c-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:25.691768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.691732    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn" event={"ID":"ec016bee-4fb5-4681-9ec6-7b9633d42af6","Type":"ContainerStarted","Data":"2e8126f42a7888af9af36c48a5a833f4484b6f311453f3b3f2a8a3ff13095fca"}
Oct 30 21:20:25.692021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.691793    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn" event={"ID":"ec016bee-4fb5-4681-9ec6-7b9633d42af6","Type":"ContainerStarted","Data":"efa8bfa4d9937552dfc33837775bebc70fe5bd9f8744532500d08980bdd5f924"}
Oct 30 21:20:25.692021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.691835    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:25.693161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.693128    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5" event={"ID":"348cbb0a-d675-4025-bc1e-b13afd70dd5c","Type":"ContainerDied","Data":"7f1bfa76e8b45fdd83fde1504456c0bde2798a5fcda1169799191f961e2dff6e"}
Oct 30 21:20:25.693284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.693173    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"
Oct 30 21:20:25.693284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.693188    4633 scope.go:122] "RemoveContainer" containerID="72cb2f71eb94e82620e7f552beedaba0f512383b0dae6d45f185a378abcdbe5a"
Oct 30 21:20:25.714115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.714071    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn" podStartSLOduration=2.1915057 podStartE2EDuration="2.714058257s" totalImagesPullingTime="522.552557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:25.711870108 +0000 UTC m=+1134.061597504" watchObservedRunningTime="2026-10-30 21:20:25.714058257 +0000 UTC m=+1134.063785700"
Oct 30 21:20:25.730493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.730468    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"]
Oct 30 21:20:25.741732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.741705    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e494a9-66f549c654-xk8m5"]
Oct 30 21:20:25.792992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:25.792964    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348cbb0a-d675-4025-bc1e-b13afd70dd5c" path="/var/lib/kubelet/pods/348cbb0a-d675-4025-bc1e-b13afd70dd5c/volumes"
Oct 30 21:20:27.253532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:27.253500    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:27.281222 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:27.281191    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:29.301947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:29.301920    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:29.302440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:29.301988    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:31.709745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:31.709713    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:20:32.129562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:32.129467    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"]
Oct 30 21:20:32.155981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:32.155930    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"]
Oct 30 21:20:33.898478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.898432    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"]
Oct 30 21:20:33.899623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.899599    4633 cpu_manager.go:414] "containerMap: removing container" podUID="348cbb0a-d675-4025-bc1e-b13afd70dd5c" containerName="maas-api"
Oct 30 21:20:33.899623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.899624    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348cbb0a-d675-4025-bc1e-b13afd70dd5c" containerName="maas-api"
Oct 30 21:20:33.899877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.899835    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="348cbb0a-d675-4025-bc1e-b13afd70dd5c" containerName="maas-api"
Oct 30 21:20:33.907484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.907455    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.917354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.916911    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2a55fb-openshift-default-dockercfg-p82lv\""
Oct 30 21:20:33.925494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.925467    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"]
Oct 30 21:20:33.960285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960249    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-istio-token\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.960546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960392    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a2699d-77ea-4e27-ac72-8428c096e798-istiod-ca-cert\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.960546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960486    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-credential-socket\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.960660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960559    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-socket\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.960732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960662    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-envoy\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.960775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960735    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a2699d-77ea-4e27-ac72-8428c096e798-istio-podinfo\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.960871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960852    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtjx\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-kube-api-access-mxtjx\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.960916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960889    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-certs\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:33.960981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:33.960967    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-data\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095290    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a2699d-77ea-4e27-ac72-8428c096e798-istiod-ca-cert\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095367    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-credential-socket\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095413    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-socket\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095500    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-envoy\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095556    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a2699d-77ea-4e27-ac72-8428c096e798-istio-podinfo\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095711    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtjx\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-kube-api-access-mxtjx\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095747    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-certs\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095856    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-data\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.095906    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-istio-token\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.097112    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a2699d-77ea-4e27-ac72-8428c096e798-istiod-ca-cert\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.097395    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-credential-socket\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.101754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.097636    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-socket\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.153741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.149430    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-data\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.153741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.149652    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-certs\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.208700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.200104    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-envoy\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.208700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.203067    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtjx\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-kube-api-access-mxtjx\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.208700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.206482    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-istio-token\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.216693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.213307    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a2699d-77ea-4e27-ac72-8428c096e798-istio-podinfo\") pod \"e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") " pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.240695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.236422    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:34.652860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.652830    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"]
Oct 30 21:20:34.654122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.654080    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:34.654259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.654186    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:34.654259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.654232    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:34.766370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.766333    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"]
Oct 30 21:20:34.798545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.798268    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"]
Oct 30 21:20:34.897707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.897634    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" event={"ID":"41a2699d-77ea-4e27-ac72-8428c096e798","Type":"ContainerStarted","Data":"81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd"}
Oct 30 21:20:34.897883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.897733    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" event={"ID":"41a2699d-77ea-4e27-ac72-8428c096e798","Type":"ContainerStarted","Data":"2f1cf6cd7b80e70b3981a9821897f1a5dbc05571459994786c5898a9610d88ee"}
Oct 30 21:20:34.927850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:34.927721    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podStartSLOduration=1.927702371 podStartE2EDuration="1.927702371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:34.924128967 +0000 UTC m=+1143.273856375" watchObservedRunningTime="2026-10-30 21:20:34.927702371 +0000 UTC m=+1143.277429766"
Oct 30 21:20:35.242196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:35.242069    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:35.246142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:35.245804    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 21:20:35.246142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:35.245866    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 21:20:36.242600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:36.242573    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:36.913493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:36.913456    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:36.914462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:36.914440    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:20:37.130396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.130346    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg" podUID="cccde7d8-1dae-40bd-bd59-a3668da8f24d" containerName="payload-processing" containerID="cri-o://efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65" gracePeriod=25
Oct 30 21:20:37.156644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.156589    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t" podUID="b9439f93-cd0f-4b03-b234-e354e15436a7" containerName="payload-pre-processing" containerID="cri-o://a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c" gracePeriod=25
Oct 30 21:20:37.214729 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:37.214638    4633 logging.go:55] [core] [Channel #1410 SubChannel #1411]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.131:50052", ServerName: "10.133.0.131:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.131:50052: connect: connection refused"
Oct 30 21:20:37.279277 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:37.279257    4633 logging.go:55] [core] [Channel #1412 SubChannel #1413]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.130:50052", ServerName: "10.133.0.130:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.130:50052: connect: connection refused"
Oct 30 21:20:37.693029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.693003    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:37.764530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.764501    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:37.787452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.787419    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9439f93-cd0f-4b03-b234-e354e15436a7-kube-api-access-jb47f\" (UniqueName: \"kubernetes.io/projected/b9439f93-cd0f-4b03-b234-e354e15436a7-kube-api-access-jb47f\") pod \"b9439f93-cd0f-4b03-b234-e354e15436a7\" (UID: \"b9439f93-cd0f-4b03-b234-e354e15436a7\") "
Oct 30 21:20:37.787611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.787493    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9439f93-cd0f-4b03-b234-e354e15436a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9439f93-cd0f-4b03-b234-e354e15436a7-plugins-config-volume\") pod \"b9439f93-cd0f-4b03-b234-e354e15436a7\" (UID: \"b9439f93-cd0f-4b03-b234-e354e15436a7\") "
Oct 30 21:20:37.787658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.787618    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cccde7d8-1dae-40bd-bd59-a3668da8f24d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccde7d8-1dae-40bd-bd59-a3668da8f24d-plugins-config-volume\") pod \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\" (UID: \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\") "
Oct 30 21:20:37.787745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.787715    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cccde7d8-1dae-40bd-bd59-a3668da8f24d-kube-api-access-fhcs7\" (UniqueName: \"kubernetes.io/projected/cccde7d8-1dae-40bd-bd59-a3668da8f24d-kube-api-access-fhcs7\") pod \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\" (UID: \"cccde7d8-1dae-40bd-bd59-a3668da8f24d\") "
Oct 30 21:20:37.788253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.788223    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccde7d8-1dae-40bd-bd59-a3668da8f24d-plugins-config-volume" pod "cccde7d8-1dae-40bd-bd59-a3668da8f24d" (UID: "cccde7d8-1dae-40bd-bd59-a3668da8f24d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:37.788253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.788223    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9439f93-cd0f-4b03-b234-e354e15436a7-plugins-config-volume" pod "b9439f93-cd0f-4b03-b234-e354e15436a7" (UID: "b9439f93-cd0f-4b03-b234-e354e15436a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:37.788514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.788492    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9439f93-cd0f-4b03-b234-e354e15436a7-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:37.788566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.788520    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccde7d8-1dae-40bd-bd59-a3668da8f24d-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:37.790731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.790702    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9439f93-cd0f-4b03-b234-e354e15436a7-kube-api-access-jb47f" pod "b9439f93-cd0f-4b03-b234-e354e15436a7" (UID: "b9439f93-cd0f-4b03-b234-e354e15436a7"). InnerVolumeSpecName "kube-api-access-jb47f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:37.791041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.791020    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccde7d8-1dae-40bd-bd59-a3668da8f24d-kube-api-access-fhcs7" pod "cccde7d8-1dae-40bd-bd59-a3668da8f24d" (UID: "cccde7d8-1dae-40bd-bd59-a3668da8f24d"). InnerVolumeSpecName "kube-api-access-fhcs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:37.890415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.890323    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhcs7\" (UniqueName: \"kubernetes.io/projected/cccde7d8-1dae-40bd-bd59-a3668da8f24d-kube-api-access-fhcs7\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:37.890415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.890357    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb47f\" (UniqueName: \"kubernetes.io/projected/b9439f93-cd0f-4b03-b234-e354e15436a7-kube-api-access-jb47f\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:37.923556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.923516    4633 generic.go:356] "Generic (PLEG): container finished" podID="cccde7d8-1dae-40bd-bd59-a3668da8f24d" containerID="efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65" exitCode=0
Oct 30 21:20:37.923756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.923568    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"
Oct 30 21:20:37.923756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.923589    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg" event={"ID":"cccde7d8-1dae-40bd-bd59-a3668da8f24d","Type":"ContainerDied","Data":"efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65"}
Oct 30 21:20:37.923756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.923643    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg" event={"ID":"cccde7d8-1dae-40bd-bd59-a3668da8f24d","Type":"ContainerDied","Data":"ca37ef0c6c8b35d0ed0a207e0d28e1102e2d769b715e8b6d2edecf64bdaece44"}
Oct 30 21:20:37.923756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.923693    4633 scope.go:122] "RemoveContainer" containerID="efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65"
Oct 30 21:20:37.925203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.925179    4633 generic.go:356] "Generic (PLEG): container finished" podID="b9439f93-cd0f-4b03-b234-e354e15436a7" containerID="a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c" exitCode=0
Oct 30 21:20:37.925329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.925215    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t" event={"ID":"b9439f93-cd0f-4b03-b234-e354e15436a7","Type":"ContainerDied","Data":"a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c"}
Oct 30 21:20:37.925329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.925264    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t" event={"ID":"b9439f93-cd0f-4b03-b234-e354e15436a7","Type":"ContainerDied","Data":"62159a013f86ac7d5fc71a0c30bfc320726b2d01f7c96decc628cb7931e14438"}
Oct 30 21:20:37.925453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.925284    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"
Oct 30 21:20:37.935083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.935064    4633 scope.go:122] "RemoveContainer" containerID="efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65"
Oct 30 21:20:37.935340 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:37.935321    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65\": container with ID starting with efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65 not found: ID does not exist" containerID="efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65"
Oct 30 21:20:37.935386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.935347    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65"} err="failed to get container status \"efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65\": rpc error: code = NotFound desc = could not find container \"efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65\": container with ID starting with efef6923cd18500139a99f8b20b627e656f2f9464694a76dad47d9e34004db65 not found: ID does not exist"
Oct 30 21:20:37.935386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.935364    4633 scope.go:122] "RemoveContainer" containerID="a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c"
Oct 30 21:20:37.945234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.945210    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"]
Oct 30 21:20:37.946070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.946053    4633 scope.go:122] "RemoveContainer" containerID="a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c"
Oct 30 21:20:37.946389 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:37.946366    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c\": container with ID starting with a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c not found: ID does not exist" containerID="a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c"
Oct 30 21:20:37.946446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.946400    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c"} err="failed to get container status \"a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c\": rpc error: code = NotFound desc = could not find container \"a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c\": container with ID starting with a43011dd20e4e35e6cd9675eb823e9a1a341a60a78beb1c9dcdb1dd0579e060c not found: ID does not exist"
Oct 30 21:20:37.949649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.949631    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg"]
Oct 30 21:20:37.962739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.962712    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"]
Oct 30 21:20:37.967866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:37.967848    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t"]
Oct 30 21:20:38.214790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.214746    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-64858ffc5f-p5rfg" podUID="cccde7d8-1dae-40bd-bd59-a3668da8f24d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.131:50052\" within 1s: context deadline exceeded"
Oct 30 21:20:38.279829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.279777    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-7867f698dc-j5k5t" podUID="b9439f93-cd0f-4b03-b234-e354e15436a7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.130:50052\" within 1s: context deadline exceeded"
Oct 30 21:20:38.455586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.455550    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"]
Oct 30 21:20:38.456284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.456268    4633 cpu_manager.go:414] "containerMap: removing container" podUID="cccde7d8-1dae-40bd-bd59-a3668da8f24d" containerName="payload-processing"
Oct 30 21:20:38.456354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.456286    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cccde7d8-1dae-40bd-bd59-a3668da8f24d" containerName="payload-processing"
Oct 30 21:20:38.456354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.456324    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b9439f93-cd0f-4b03-b234-e354e15436a7" containerName="payload-pre-processing"
Oct 30 21:20:38.456354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.456329    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9439f93-cd0f-4b03-b234-e354e15436a7" containerName="payload-pre-processing"
Oct 30 21:20:38.456448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.456432    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9439f93-cd0f-4b03-b234-e354e15436a7" containerName="payload-pre-processing"
Oct 30 21:20:38.456448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.456444    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cccde7d8-1dae-40bd-bd59-a3668da8f24d" containerName="payload-processing"
Oct 30 21:20:38.461750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.461725    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:38.463197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.463174    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"]
Oct 30 21:20:38.467897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.467870    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:38.470472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.470447    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"]
Oct 30 21:20:38.471017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.470996    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a55fb\""
Oct 30 21:20:38.474661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.474641    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2a55fb-dockercfg-rkqqq\""
Oct 30 21:20:38.478861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.477391    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"]
Oct 30 21:20:38.499743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.499191    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdl2t\" (UniqueName: \"kubernetes.io/projected/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-kube-api-access-wdl2t\") pod \"payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v\" (UID: \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:38.499743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.499253    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v\" (UID: \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:38.499743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.499294    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvhs\" (UniqueName: \"kubernetes.io/projected/1d520039-aae0-4404-b51f-4e1d0477692a-kube-api-access-kzvhs\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn\" (UID: \"1d520039-aae0-4404-b51f-4e1d0477692a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:38.499743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.499601    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d520039-aae0-4404-b51f-4e1d0477692a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn\" (UID: \"1d520039-aae0-4404-b51f-4e1d0477692a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:38.607705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.602746    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d520039-aae0-4404-b51f-4e1d0477692a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn\" (UID: \"1d520039-aae0-4404-b51f-4e1d0477692a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:38.607705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.603010    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdl2t\" (UniqueName: \"kubernetes.io/projected/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-kube-api-access-wdl2t\") pod \"payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v\" (UID: \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:38.607705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.603038    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v\" (UID: \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:38.607705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.603076    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvhs\" (UniqueName: \"kubernetes.io/projected/1d520039-aae0-4404-b51f-4e1d0477692a-kube-api-access-kzvhs\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn\" (UID: \"1d520039-aae0-4404-b51f-4e1d0477692a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:38.607705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.605199    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d520039-aae0-4404-b51f-4e1d0477692a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn\" (UID: \"1d520039-aae0-4404-b51f-4e1d0477692a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:38.676710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.664810    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvhs\" (UniqueName: \"kubernetes.io/projected/1d520039-aae0-4404-b51f-4e1d0477692a-kube-api-access-kzvhs\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn\" (UID: \"1d520039-aae0-4404-b51f-4e1d0477692a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:38.676710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.672020    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v\" (UID: \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:38.695447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.695361    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdl2t\" (UniqueName: \"kubernetes.io/projected/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-kube-api-access-wdl2t\") pod \"payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v\" (UID: \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:38.784156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.783999    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:38.798075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:38.796994    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:39.177560 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:39.177529    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436e8e3d_1cd0_4e50_91f1_60004acfd7f2.slice/crio-5fb28161f9de1d1d33e8e810cfcdc1dd666777aa709defe7935dc5e1aded759b WatchSource:0}: Error finding container 5fb28161f9de1d1d33e8e810cfcdc1dd666777aa709defe7935dc5e1aded759b: Status 404 returned error can't find the container with id 5fb28161f9de1d1d33e8e810cfcdc1dd666777aa709defe7935dc5e1aded759b
Oct 30 21:20:39.178877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:39.178855    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"]
Oct 30 21:20:39.185311 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:39.185276    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d520039_aae0_4404_b51f_4e1d0477692a.slice/crio-2539db671e1a0cb37cea9a3efcc093c5d5ec0f18310af0ea7346f13618cd4fb3 WatchSource:0}: Error finding container 2539db671e1a0cb37cea9a3efcc093c5d5ec0f18310af0ea7346f13618cd4fb3: Status 404 returned error can't find the container with id 2539db671e1a0cb37cea9a3efcc093c5d5ec0f18310af0ea7346f13618cd4fb3
Oct 30 21:20:39.188031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:39.188006    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"]
Oct 30 21:20:39.767755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:39.767642    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb" podUID="122c0c18-c1e4-4e4b-8597-e8d8c59c24d9" containerName="payload-processing" containerID="cri-o://d47d91b98c67eda67b58a744930ab59720a25283cc189341a26e53fb2f5fc9ac" gracePeriod=25
Oct 30 21:20:39.793379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:39.793348    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9439f93-cd0f-4b03-b234-e354e15436a7" path="/var/lib/kubelet/pods/b9439f93-cd0f-4b03-b234-e354e15436a7/volumes"
Oct 30 21:20:39.793895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:39.793870    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cccde7d8-1dae-40bd-bd59-a3668da8f24d" path="/var/lib/kubelet/pods/cccde7d8-1dae-40bd-bd59-a3668da8f24d/volumes"
Oct 30 21:20:39.799637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:39.799610    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s" podUID="afd40b3f-da73-4089-a78c-8699dd3e5077" containerName="payload-pre-processing" containerID="cri-o://da51b7e5f6b0511b77637548d5bf4d2960b8a3cea7dc63b62f50a514e724d6b7" gracePeriod=25
Oct 30 21:20:40.143820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.136372    4633 generic.go:356] "Generic (PLEG): container finished" podID="122c0c18-c1e4-4e4b-8597-e8d8c59c24d9" containerID="d47d91b98c67eda67b58a744930ab59720a25283cc189341a26e53fb2f5fc9ac" exitCode=0
Oct 30 21:20:40.143820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.136549    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb" event={"ID":"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9","Type":"ContainerDied","Data":"d47d91b98c67eda67b58a744930ab59720a25283cc189341a26e53fb2f5fc9ac"}
Oct 30 21:20:40.143820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.143561    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn" event={"ID":"1d520039-aae0-4404-b51f-4e1d0477692a","Type":"ContainerStarted","Data":"305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2"}
Oct 30 21:20:40.143820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.143647    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn" event={"ID":"1d520039-aae0-4404-b51f-4e1d0477692a","Type":"ContainerStarted","Data":"2539db671e1a0cb37cea9a3efcc093c5d5ec0f18310af0ea7346f13618cd4fb3"}
Oct 30 21:20:40.145086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.145008    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:40.149852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.148005    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v" event={"ID":"436e8e3d-1cd0-4e50-91f1-60004acfd7f2","Type":"ContainerStarted","Data":"acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399"}
Oct 30 21:20:40.149852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.148083    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v" event={"ID":"436e8e3d-1cd0-4e50-91f1-60004acfd7f2","Type":"ContainerStarted","Data":"5fb28161f9de1d1d33e8e810cfcdc1dd666777aa709defe7935dc5e1aded759b"}
Oct 30 21:20:40.149852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.148982    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:40.153880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.152783    4633 generic.go:356] "Generic (PLEG): container finished" podID="afd40b3f-da73-4089-a78c-8699dd3e5077" containerID="da51b7e5f6b0511b77637548d5bf4d2960b8a3cea7dc63b62f50a514e724d6b7" exitCode=0
Oct 30 21:20:40.153880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.152872    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s" event={"ID":"afd40b3f-da73-4089-a78c-8699dd3e5077","Type":"ContainerDied","Data":"da51b7e5f6b0511b77637548d5bf4d2960b8a3cea7dc63b62f50a514e724d6b7"}
Oct 30 21:20:40.181272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.181149    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn" podStartSLOduration=2.181130963 podStartE2EDuration="2.181130963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:40.180189166 +0000 UTC m=+1148.529916563" watchObservedRunningTime="2026-10-30 21:20:40.181130963 +0000 UTC m=+1148.530858359"
Oct 30 21:20:40.259991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.259936    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v" podStartSLOduration=2.259920156 podStartE2EDuration="2.259920156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:40.234548846 +0000 UTC m=+1148.584276244" watchObservedRunningTime="2026-10-30 21:20:40.259920156 +0000 UTC m=+1148.609647577"
Oct 30 21:20:40.503309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.503280    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:40.553453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.553428    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:40.626993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.626964    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd40b3f-da73-4089-a78c-8699dd3e5077-kube-api-access-f6566\" (UniqueName: \"kubernetes.io/projected/afd40b3f-da73-4089-a78c-8699dd3e5077-kube-api-access-f6566\") pod \"afd40b3f-da73-4089-a78c-8699dd3e5077\" (UID: \"afd40b3f-da73-4089-a78c-8699dd3e5077\") "
Oct 30 21:20:40.627199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.627003    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-plugins-config-volume\") pod \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\" (UID: \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\") "
Oct 30 21:20:40.627199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.627099    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd40b3f-da73-4089-a78c-8699dd3e5077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd40b3f-da73-4089-a78c-8699dd3e5077-plugins-config-volume\") pod \"afd40b3f-da73-4089-a78c-8699dd3e5077\" (UID: \"afd40b3f-da73-4089-a78c-8699dd3e5077\") "
Oct 30 21:20:40.627199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.627125    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-kube-api-access-x54kn\" (UniqueName: \"kubernetes.io/projected/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-kube-api-access-x54kn\") pod \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\" (UID: \"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9\") "
Oct 30 21:20:40.627521 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.627492    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-plugins-config-volume" pod "122c0c18-c1e4-4e4b-8597-e8d8c59c24d9" (UID: "122c0c18-c1e4-4e4b-8597-e8d8c59c24d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:40.627612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.627492    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd40b3f-da73-4089-a78c-8699dd3e5077-plugins-config-volume" pod "afd40b3f-da73-4089-a78c-8699dd3e5077" (UID: "afd40b3f-da73-4089-a78c-8699dd3e5077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:40.627725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.627680    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.627725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.627700    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd40b3f-da73-4089-a78c-8699dd3e5077-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.629549 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.629518    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-kube-api-access-x54kn" pod "122c0c18-c1e4-4e4b-8597-e8d8c59c24d9" (UID: "122c0c18-c1e4-4e4b-8597-e8d8c59c24d9"). InnerVolumeSpecName "kube-api-access-x54kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:40.629645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.629585    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd40b3f-da73-4089-a78c-8699dd3e5077-kube-api-access-f6566" pod "afd40b3f-da73-4089-a78c-8699dd3e5077" (UID: "afd40b3f-da73-4089-a78c-8699dd3e5077"). InnerVolumeSpecName "kube-api-access-f6566". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:40.729387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.729294    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6566\" (UniqueName: \"kubernetes.io/projected/afd40b3f-da73-4089-a78c-8699dd3e5077-kube-api-access-f6566\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.729387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:40.729330    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x54kn\" (UniqueName: \"kubernetes.io/projected/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9-kube-api-access-x54kn\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:41.169779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.165888    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"
Oct 30 21:20:41.169779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.165925    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb" event={"ID":"122c0c18-c1e4-4e4b-8597-e8d8c59c24d9","Type":"ContainerDied","Data":"556889f7f31d2f63b7f937d37f377311abcae39b1f1041019001c998524f0972"}
Oct 30 21:20:41.169779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.165995    4633 scope.go:122] "RemoveContainer" containerID="d47d91b98c67eda67b58a744930ab59720a25283cc189341a26e53fb2f5fc9ac"
Oct 30 21:20:41.172092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.172027    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"
Oct 30 21:20:41.172092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.172047    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s" event={"ID":"afd40b3f-da73-4089-a78c-8699dd3e5077","Type":"ContainerDied","Data":"14c9ea9c17fc49a3fde3368365591824c45ed22f57236aae6afd3d532745a92b"}
Oct 30 21:20:41.182313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.182287    4633 scope.go:122] "RemoveContainer" containerID="da51b7e5f6b0511b77637548d5bf4d2960b8a3cea7dc63b62f50a514e724d6b7"
Oct 30 21:20:41.210425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.210387    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"]
Oct 30 21:20:41.215184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.215158    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-6565cd65c9-65gzb"]
Oct 30 21:20:41.225232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.225195    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"]
Oct 30 21:20:41.229442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.229417    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-9455ff695-ll62s"]
Oct 30 21:20:41.793991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.793951    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122c0c18-c1e4-4e4b-8597-e8d8c59c24d9" path="/var/lib/kubelet/pods/122c0c18-c1e4-4e4b-8597-e8d8c59c24d9/volumes"
Oct 30 21:20:41.794420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:41.794404    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd40b3f-da73-4089-a78c-8699dd3e5077" path="/var/lib/kubelet/pods/afd40b3f-da73-4089-a78c-8699dd3e5077/volumes"
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.042895    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"]
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.045487    4633 cpu_manager.go:414] "containerMap: removing container" podUID="122c0c18-c1e4-4e4b-8597-e8d8c59c24d9" containerName="payload-processing"
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.045507    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122c0c18-c1e4-4e4b-8597-e8d8c59c24d9" containerName="payload-processing"
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.045557    4633 cpu_manager.go:414] "containerMap: removing container" podUID="afd40b3f-da73-4089-a78c-8699dd3e5077" containerName="payload-pre-processing"
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.045563    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd40b3f-da73-4089-a78c-8699dd3e5077" containerName="payload-pre-processing"
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.045659    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd40b3f-da73-4089-a78c-8699dd3e5077" containerName="payload-pre-processing"
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.045689    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="122c0c18-c1e4-4e4b-8597-e8d8c59c24d9" containerName="payload-processing"
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.054203    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"]
Oct 30 21:20:43.055324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.054370    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:43.085420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.085384    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60c26c-dockercfg-rt2qf\""
Oct 30 21:20:43.086768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.086743    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60c26c\""
Oct 30 21:20:43.212227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.212194    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zzz2\" (UniqueName: \"kubernetes.io/projected/399a9845-1ede-4514-9177-049998f244aa-kube-api-access-4zzz2\") pod \"payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp\" (UID: \"399a9845-1ede-4514-9177-049998f244aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:43.212398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.212241    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399a9845-1ede-4514-9177-049998f244aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp\" (UID: \"399a9845-1ede-4514-9177-049998f244aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:43.313779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.313657    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zzz2\" (UniqueName: \"kubernetes.io/projected/399a9845-1ede-4514-9177-049998f244aa-kube-api-access-4zzz2\") pod \"payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp\" (UID: \"399a9845-1ede-4514-9177-049998f244aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:43.314410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.314383    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399a9845-1ede-4514-9177-049998f244aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp\" (UID: \"399a9845-1ede-4514-9177-049998f244aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:43.328181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.315444    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399a9845-1ede-4514-9177-049998f244aa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp\" (UID: \"399a9845-1ede-4514-9177-049998f244aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:43.328415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.328181    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zzz2\" (UniqueName: \"kubernetes.io/projected/399a9845-1ede-4514-9177-049998f244aa-kube-api-access-4zzz2\") pod \"payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp\" (UID: \"399a9845-1ede-4514-9177-049998f244aa\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:43.443117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.443078    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:43.877922 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:20:43.877886    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399a9845_1ede_4514_9177_049998f244aa.slice/crio-278af5b159391fcaf6faa5bbbf521da4d79241a5be7bb530ac6af694ec25ccc4 WatchSource:0}: Error finding container 278af5b159391fcaf6faa5bbbf521da4d79241a5be7bb530ac6af694ec25ccc4: Status 404 returned error can't find the container with id 278af5b159391fcaf6faa5bbbf521da4d79241a5be7bb530ac6af694ec25ccc4
Oct 30 21:20:43.885628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:43.885602    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"]
Oct 30 21:20:44.208125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:44.204265    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp" event={"ID":"399a9845-1ede-4514-9177-049998f244aa","Type":"ContainerStarted","Data":"278af5b159391fcaf6faa5bbbf521da4d79241a5be7bb530ac6af694ec25ccc4"}
Oct 30 21:20:46.279254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:46.279210    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp" event={"ID":"399a9845-1ede-4514-9177-049998f244aa","Type":"ContainerStarted","Data":"2bbac29f8bcbd7faac1b189a95d28b37ab43e437327ef8bb65095f6a5b5802b5"}
Oct 30 21:20:46.281092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:46.281066    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:50.923243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:50.923170    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp" podStartSLOduration=7.252368228 podStartE2EDuration="8.923147068s" totalImagesPullingTime="1.67077884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:46.310154459 +0000 UTC m=+1154.659881859" watchObservedRunningTime="2026-10-30 21:20:50.923147068 +0000 UTC m=+1159.272874465"
Oct 30 21:20:50.925058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:50.925019    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"]
Oct 30 21:20:50.925937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:50.925905    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy" containerID="cri-o://3bb1dd6dc74bb4311b3cc1f6c82089a412b1c62696ef554422d7b937ac797322" gracePeriod=30
Oct 30 21:20:52.186414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.186379    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:20:52.186855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.186470    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:20:52.214045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.214009    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"]
Oct 30 21:20:52.262033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.261992    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.267684 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.267632    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"]
Oct 30 21:20:52.271582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.271554    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dbc08f-openshift-default-dockercfg-q4ml7\""
Oct 30 21:20:52.373552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373517    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4cd\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-kube-api-access-8h4cd\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.373795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373567    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-token\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.373795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373600    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01d8427-ce69-4098-b6c9-cf2eb7530091-istiod-ca-cert\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.373795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373626    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-socket\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.373795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373689    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-credential-socket\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.373795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373737    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-envoy\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.373795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373772    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-data\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.374217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373887    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-certs\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.374217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.373932    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-podinfo\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.490769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.476745    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4cd\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-kube-api-access-8h4cd\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.491280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.491241    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-token\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.491479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.491463    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01d8427-ce69-4098-b6c9-cf2eb7530091-istiod-ca-cert\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.492757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.492733    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-socket\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.492977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.492959    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-credential-socket\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.496727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.493121    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-envoy\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.515460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.515427    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-data\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.515814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.492645    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01d8427-ce69-4098-b6c9-cf2eb7530091-istiod-ca-cert\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.515814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.513984    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-credential-socket\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.515814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.515622    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4cd\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-kube-api-access-8h4cd\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.515814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.513620    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-socket\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.516074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.515927    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-data\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.516448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.516426    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-certs\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.516992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.516967    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-podinfo\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.518314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.516862    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-certs\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.547171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.547088    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-envoy\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.548119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.548065    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-podinfo\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.552356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.552333    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-token\") pod \"e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") " pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:52.609887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:52.609854    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:53.160110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:53.158714    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"]
Oct 30 21:20:53.165117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:53.165071    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:53.165336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:53.165309    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:53.166117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:53.166092    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:53.456778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:53.456552    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" event={"ID":"f01d8427-ce69-4098-b6c9-cf2eb7530091","Type":"ContainerStarted","Data":"b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b"}
Oct 30 21:20:53.456778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:53.456616    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" event={"ID":"f01d8427-ce69-4098-b6c9-cf2eb7530091","Type":"ContainerStarted","Data":"cb46d44ae9a1fc968bcde9f354772dd4569de8edfd7541d5e5957d4516a87d25"}
Oct 30 21:20:53.479456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:53.479404    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podStartSLOduration=1.479390188 podStartE2EDuration="1.479390188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:53.475041036 +0000 UTC m=+1161.824768432" watchObservedRunningTime="2026-10-30 21:20:53.479390188 +0000 UTC m=+1161.829117583"
Oct 30 21:20:53.610869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:53.610774    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:54.466902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:54.466869    4633 generic.go:356] "Generic (PLEG): container finished" podID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerID="534b1fa378bfe125e4d4b529ead2b1c62a51ce3ed36771f18b89393062f1dfb5" exitCode=0
Oct 30 21:20:54.467436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:54.466947    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" event={"ID":"4f416feb-13be-4a2c-b2bc-9999484d1d56","Type":"ContainerDied","Data":"534b1fa378bfe125e4d4b529ead2b1c62a51ce3ed36771f18b89393062f1dfb5"}
Oct 30 21:20:54.467534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:54.467517    4633 scope.go:122] "RemoveContainer" containerID="534b1fa378bfe125e4d4b529ead2b1c62a51ce3ed36771f18b89393062f1dfb5"
Oct 30 21:20:54.469370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:54.469341    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:54.469450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:54.469414    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:54.469491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:54.469448    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:20:54.625776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:54.625745    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:55.041296 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:20:55.041263    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:20:55.478767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:55.478725    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" event={"ID":"4f416feb-13be-4a2c-b2bc-9999484d1d56","Type":"ContainerStarted","Data":"4efbb86417c450fe9a827199cef79c90a604a34ca660c1e4e843551e1ad1978f"}
Oct 30 21:20:55.479205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:55.479184    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:55.480336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:55.480313    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:20:55.733754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:55.729948    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:55.884481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:55.884442    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:20:56.223625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.219098    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"]
Oct 30 21:20:56.233725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.226565    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.278790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.277711    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4c6998\"/\"openshift-service-ca.crt\""
Oct 30 21:20:56.278790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.277856    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4c6998\"/\"kube-root-ca.crt\""
Oct 30 21:20:56.278790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.277966    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4c6998\"/\"default-dockercfg-j5tfc\""
Oct 30 21:20:56.278790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.278045    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4c6998\"/\"ipp-route-4c6998-kserve-self-signed-certs\""
Oct 30 21:20:56.303439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.303364    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"]
Oct 30 21:20:56.308055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.307984    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:20:56.309011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.308977    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:20:56.309195 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.309176    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:20:56.400702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.378977    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-model-cache\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.400702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.379042    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tmp-dir\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.400702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.379100    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hr5\" (UniqueName: \"kubernetes.io/projected/2d19f591-0fab-472f-8a17-4bd876fb6fc5-kube-api-access-j9hr5\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.400702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.379205    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-home\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.400702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.379257    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tls-certs\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.400702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.379296    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-dshm\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.649482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.648187    4633 generic.go:356] "Generic (PLEG): container finished" podID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerID="3bb1dd6dc74bb4311b3cc1f6c82089a412b1c62696ef554422d7b937ac797322" exitCode=0
Oct 30 21:20:56.649482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.648883    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" event={"ID":"0cd236d7-7f72-49a1-850c-642a23a4ef34","Type":"ContainerDied","Data":"3bb1dd6dc74bb4311b3cc1f6c82089a412b1c62696ef554422d7b937ac797322"}
Oct 30 21:20:56.649482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.648940    4633 scope.go:122] "RemoveContainer" containerID="4a2950a4835efb8ddf42a5c2cd4a5f489074b719429ed6bc4e4e5912e1b0f2d5"
Oct 30 21:20:56.650422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.650397    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-dshm\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.650798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.650776    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-model-cache\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.650960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.650946    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tmp-dir\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.652491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.651502    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tmp-dir\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.652491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.651766    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-model-cache\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.652491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.651888    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hr5\" (UniqueName: \"kubernetes.io/projected/2d19f591-0fab-472f-8a17-4bd876fb6fc5-kube-api-access-j9hr5\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.652491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.652000    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-home\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.652491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.652061    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tls-certs\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.653588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.653565    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-home\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.814857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.814825    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tls-certs\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.879914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.874725    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-dshm\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:56.907117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:56.904835    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hr5\" (UniqueName: \"kubernetes.io/projected/2d19f591-0fab-472f-8a17-4bd876fb6fc5-kube-api-access-j9hr5\") pod \"ipp-route-4c6998-kserve-65d9fffbb9-bx8dn\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") " pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:57.215464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.211000    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:57.333205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.331158    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:20:57.333205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.331223    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:20:57.545067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.545025    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:20:57.648251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.648223    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"]
Oct 30 21:20:57.661340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.660940    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-envoy\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.661340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.661009    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-podinfo\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.661340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.661075    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-socket\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.661340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.661221    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-certs\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.661340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.661318    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-credential-socket\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.661952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.661372    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd236d7-7f72-49a1-850c-642a23a4ef34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd236d7-7f72-49a1-850c-642a23a4ef34-istiod-ca-cert\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.661952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.661496    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-token\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-token\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.661952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.661541    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-data\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.661952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.661610    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-kube-api-access-526p9\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-kube-api-access-526p9\") pod \"0cd236d7-7f72-49a1-850c-642a23a4ef34\" (UID: \"0cd236d7-7f72-49a1-850c-642a23a4ef34\") "
Oct 30 21:20:57.662148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.662102    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd236d7-7f72-49a1-850c-642a23a4ef34-istiod-ca-cert" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:57.662476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.662251    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-socket" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.662476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.662278    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cd236d7-7f72-49a1-850c-642a23a4ef34-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.662476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.662446    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-data" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.662614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.662558    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-credential-socket" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.662651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.662626    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-certs" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.663937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.663912    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-envoy" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.665065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.665041    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-podinfo" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:20:57.665324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.665288    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-token" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:57.665831 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.665806    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn" event={"ID":"0cd236d7-7f72-49a1-850c-642a23a4ef34","Type":"ContainerDied","Data":"3473f487588dd563a709f9188ac05069d528b7970b9ee4564de4399dad335d9a"}
Oct 30 21:20:57.665927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.665861    4633 scope.go:122] "RemoveContainer" containerID="3bb1dd6dc74bb4311b3cc1f6c82089a412b1c62696ef554422d7b937ac797322"
Oct 30 21:20:57.666433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.666414    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"
Oct 30 21:20:57.667045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.667017    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-kube-api-access-526p9" pod "0cd236d7-7f72-49a1-850c-642a23a4ef34" (UID: "0cd236d7-7f72-49a1-850c-642a23a4ef34"). InnerVolumeSpecName "kube-api-access-526p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:57.764245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.764217    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.764245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.764246    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-526p9\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-kube-api-access-526p9\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.764434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.764260    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.764434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.764287    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.764434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.764313    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.764434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.764329    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.764434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.764344    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cd236d7-7f72-49a1-850c-642a23a4ef34-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.764434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.764358    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cd236d7-7f72-49a1-850c-642a23a4ef34-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.996022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:57.995941    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"]
Oct 30 21:20:58.002103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.002074    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-4k2gn"]
Oct 30 21:20:58.209656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.208872    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"]
Oct 30 21:20:58.317766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.309931    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:20:58.317766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.311922    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:20:58.317766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.311976    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:20:58.392708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.363339    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"]
Oct 30 21:20:58.762693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.762637    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn" event={"ID":"2d19f591-0fab-472f-8a17-4bd876fb6fc5","Type":"ContainerStarted","Data":"4827b2cedbe1f210eb3036372d6e0139be2bdd49ac03f8e7f0bd277331398b77"}
Oct 30 21:20:58.762693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.762694    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn" event={"ID":"2d19f591-0fab-472f-8a17-4bd876fb6fc5","Type":"ContainerStarted","Data":"e18e618f4cfd7ce4184a73796080039eacb3187682a0fff860e291e06a0fd24a"}
Oct 30 21:20:58.783903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:58.783570    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn" podStartSLOduration=2.783557137 podStartE2EDuration="2.783557137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:58.779557819 +0000 UTC m=+1167.129285216" watchObservedRunningTime="2026-10-30 21:20:58.783557137 +0000 UTC m=+1167.133284533"
Oct 30 21:20:59.227790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:59.222878    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:20:59.309725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:59.308415    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:20:59.309725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:59.308462    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:20:59.793072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:20:59.793040    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" path="/var/lib/kubelet/pods/0cd236d7-7f72-49a1-850c-642a23a4ef34/volumes"
Oct 30 21:21:00.308774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:00.308742    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:21:00.309473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:00.309445    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:21:01.308809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:01.308774    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:21:01.309316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:01.308829    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:21:02.308737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:02.308695    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 21:21:02.308930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:02.308757    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 21:21:03.209811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.209768    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v" podUID="436e8e3d-1cd0-4e50-91f1-60004acfd7f2" containerName="payload-processing" containerID="cri-o://acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399" gracePeriod=25
Oct 30 21:21:03.323694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.312836    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:21:03.340030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.339999    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:21:03.364531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.364463    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn" podUID="1d520039-aae0-4404-b51f-4e1d0477692a" containerName="payload-pre-processing" containerID="cri-o://305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2" gracePeriod=25
Oct 30 21:21:03.768823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.768347    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:21:03.853951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.853923    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-plugins-config-volume\") pod \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\" (UID: \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\") "
Oct 30 21:21:03.854159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.854064    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-kube-api-access-wdl2t\" (UniqueName: \"kubernetes.io/projected/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-kube-api-access-wdl2t\") pod \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\" (UID: \"436e8e3d-1cd0-4e50-91f1-60004acfd7f2\") "
Oct 30 21:21:03.854375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.854349    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-plugins-config-volume" pod "436e8e3d-1cd0-4e50-91f1-60004acfd7f2" (UID: "436e8e3d-1cd0-4e50-91f1-60004acfd7f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:03.855006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.854980    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:03.856454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.856427    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-kube-api-access-wdl2t" pod "436e8e3d-1cd0-4e50-91f1-60004acfd7f2" (UID: "436e8e3d-1cd0-4e50-91f1-60004acfd7f2"). InnerVolumeSpecName "kube-api-access-wdl2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:03.883708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.883649    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:21:03.893198 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.893173    4633 generic.go:356] "Generic (PLEG): container finished" podID="436e8e3d-1cd0-4e50-91f1-60004acfd7f2" containerID="acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399" exitCode=0
Oct 30 21:21:03.893332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.893221    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"
Oct 30 21:21:03.893332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.893258    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v" event={"ID":"436e8e3d-1cd0-4e50-91f1-60004acfd7f2","Type":"ContainerDied","Data":"acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399"}
Oct 30 21:21:03.893332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.893314    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v" event={"ID":"436e8e3d-1cd0-4e50-91f1-60004acfd7f2","Type":"ContainerDied","Data":"5fb28161f9de1d1d33e8e810cfcdc1dd666777aa709defe7935dc5e1aded759b"}
Oct 30 21:21:03.893478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.893338    4633 scope.go:122] "RemoveContainer" containerID="acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399"
Oct 30 21:21:03.894871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.894848    4633 generic.go:356] "Generic (PLEG): container finished" podID="1d520039-aae0-4404-b51f-4e1d0477692a" containerID="305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2" exitCode=0
Oct 30 21:21:03.894977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.894889    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"
Oct 30 21:21:03.894977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.894937    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn" event={"ID":"1d520039-aae0-4404-b51f-4e1d0477692a","Type":"ContainerDied","Data":"305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2"}
Oct 30 21:21:03.894977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.894966    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn" event={"ID":"1d520039-aae0-4404-b51f-4e1d0477692a","Type":"ContainerDied","Data":"2539db671e1a0cb37cea9a3efcc093c5d5ec0f18310af0ea7346f13618cd4fb3"}
Oct 30 21:21:03.907240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.907223    4633 scope.go:122] "RemoveContainer" containerID="acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399"
Oct 30 21:21:03.907629 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:03.907603    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399\": container with ID starting with acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399 not found: ID does not exist" containerID="acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399"
Oct 30 21:21:03.907816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.907643    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399"} err="failed to get container status \"acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399\": rpc error: code = NotFound desc = could not find container \"acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399\": container with ID starting with acd9d1f09264ac89092fc35eebd2754bc1d9557db25b2147c7e3ccfcb355a399 not found: ID does not exist"
Oct 30 21:21:03.907816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.907663    4633 scope.go:122] "RemoveContainer" containerID="305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2"
Oct 30 21:21:03.945991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.945959    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"]
Oct 30 21:21:03.950781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.950750    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-7d5cc9b995-g877v"]
Oct 30 21:21:03.955605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.955464    4633 scope.go:122] "RemoveContainer" containerID="305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2"
Oct 30 21:21:03.956273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.955892    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d520039-aae0-4404-b51f-4e1d0477692a-kube-api-access-kzvhs\" (UniqueName: \"kubernetes.io/projected/1d520039-aae0-4404-b51f-4e1d0477692a-kube-api-access-kzvhs\") pod \"1d520039-aae0-4404-b51f-4e1d0477692a\" (UID: \"1d520039-aae0-4404-b51f-4e1d0477692a\") "
Oct 30 21:21:03.956273 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:03.955934    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2\": container with ID starting with 305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2 not found: ID does not exist" containerID="305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2"
Oct 30 21:21:03.956273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.955971    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2"} err="failed to get container status \"305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2\": rpc error: code = NotFound desc = could not find container \"305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2\": container with ID starting with 305f1ab5072024692560ed5a3c181721bc68c35b79e34fad299d9cbc825631b2 not found: ID does not exist"
Oct 30 21:21:03.956273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.956008    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d520039-aae0-4404-b51f-4e1d0477692a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d520039-aae0-4404-b51f-4e1d0477692a-plugins-config-volume\") pod \"1d520039-aae0-4404-b51f-4e1d0477692a\" (UID: \"1d520039-aae0-4404-b51f-4e1d0477692a\") "
Oct 30 21:21:03.956770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.956742    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d520039-aae0-4404-b51f-4e1d0477692a-plugins-config-volume" pod "1d520039-aae0-4404-b51f-4e1d0477692a" (UID: "1d520039-aae0-4404-b51f-4e1d0477692a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:03.956899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.956879    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdl2t\" (UniqueName: \"kubernetes.io/projected/436e8e3d-1cd0-4e50-91f1-60004acfd7f2-kube-api-access-wdl2t\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:03.959279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:03.959258    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d520039-aae0-4404-b51f-4e1d0477692a-kube-api-access-kzvhs" pod "1d520039-aae0-4404-b51f-4e1d0477692a" (UID: "1d520039-aae0-4404-b51f-4e1d0477692a"). InnerVolumeSpecName "kube-api-access-kzvhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:04.058688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:04.058602    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzvhs\" (UniqueName: \"kubernetes.io/projected/1d520039-aae0-4404-b51f-4e1d0477692a-kube-api-access-kzvhs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:04.058688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:04.058626    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d520039-aae0-4404-b51f-4e1d0477692a-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:04.240479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:04.240445    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"]
Oct 30 21:21:04.246786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:04.246747    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-65ddbd85c5-wgnjn"]
Oct 30 21:21:05.839750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:05.839184    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d520039-aae0-4404-b51f-4e1d0477692a" path="/var/lib/kubelet/pods/1d520039-aae0-4404-b51f-4e1d0477692a/volumes"
Oct 30 21:21:05.840318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:05.840037    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436e8e3d-1cd0-4e50-91f1-60004acfd7f2" path="/var/lib/kubelet/pods/436e8e3d-1cd0-4e50-91f1-60004acfd7f2/volumes"
Oct 30 21:21:07.219410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:07.219377    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:21:07.233149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:07.233115    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:21:07.967741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:07.959496    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:21:07.967741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:07.959557    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:21:07.987080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:07.987045    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:21:09.985753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:09.985627    4633 generic.go:356] "Generic (PLEG): container finished" podID="41a2699d-77ea-4e27-ac72-8428c096e798" containerID="81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd" exitCode=0
Oct 30 21:21:09.985753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:09.985698    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" event={"ID":"41a2699d-77ea-4e27-ac72-8428c096e798","Type":"ContainerDied","Data":"81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd"}
Oct 30 21:21:09.986205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:09.986184    4633 scope.go:122] "RemoveContainer" containerID="81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd"
Oct 30 21:21:09.988050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:09.988016    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:09.988162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:09.988109    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:09.988215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:09.988156    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:10.997078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:10.997046    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" event={"ID":"41a2699d-77ea-4e27-ac72-8428c096e798","Type":"ContainerStarted","Data":"df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916"}
Oct 30 21:21:11.238041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:11.238009    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:21:11.238334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:11.238152    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 21:21:11.238438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:11.238366    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 21:21:12.238188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:12.238141    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 21:21:12.238782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:12.238202    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 21:21:13.238694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:13.238286    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 21:21:13.238694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:13.238362    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 21:21:14.238135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:14.238099    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 21:21:14.238332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:14.238158    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 21:21:15.238099 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:15.238063    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 21:21:15.238524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:15.238118    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 21:21:16.245695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:16.238154    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Oct 30 21:21:16.245695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:16.238236    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Oct 30 21:21:17.241905 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:17.241871    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:21:17.242320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:17.242298    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:21:17.243208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:17.243188    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:21:17.704160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:17.704128    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:21:18.991760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:18.858747    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"]
Oct 30 21:21:18.991760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:18.988537    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5" podUID="4074b6d4-ca47-47ba-9677-0e0b9ba2099d" containerName="maas-api" containerID="cri-o://6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5" gracePeriod=30
Oct 30 21:21:19.155699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:19.152819    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5" podUID="4074b6d4-ca47-47ba-9677-0e0b9ba2099d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.126:8443/health\": dial tcp 10.133.0.126:8443: connect: connection refused"
Oct 30 21:21:19.903208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:19.903165    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:21:20.030657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.029218    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-openshift-service-ca\") pod \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") "
Oct 30 21:21:20.030657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.029304    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-metrics-certs\") pod \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") "
Oct 30 21:21:20.030657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.029432    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-maas-api-tls\") pod \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") "
Oct 30 21:21:20.030657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.029632    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-kube-api-access-qsb4p\" (UniqueName: \"kubernetes.io/projected/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-kube-api-access-qsb4p\") pod \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\" (UID: \"4074b6d4-ca47-47ba-9677-0e0b9ba2099d\") "
Oct 30 21:21:20.032320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.032291    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-openshift-service-ca" pod "4074b6d4-ca47-47ba-9677-0e0b9ba2099d" (UID: "4074b6d4-ca47-47ba-9677-0e0b9ba2099d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:20.044873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.044839    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-maas-api-tls" pod "4074b6d4-ca47-47ba-9677-0e0b9ba2099d" (UID: "4074b6d4-ca47-47ba-9677-0e0b9ba2099d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:20.050852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.050815    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-metrics-certs" pod "4074b6d4-ca47-47ba-9677-0e0b9ba2099d" (UID: "4074b6d4-ca47-47ba-9677-0e0b9ba2099d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:20.051203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.051175    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-kube-api-access-qsb4p" pod "4074b6d4-ca47-47ba-9677-0e0b9ba2099d" (UID: "4074b6d4-ca47-47ba-9677-0e0b9ba2099d"). InnerVolumeSpecName "kube-api-access-qsb4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:20.103400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.103308    4633 generic.go:356] "Generic (PLEG): container finished" podID="4074b6d4-ca47-47ba-9677-0e0b9ba2099d" containerID="6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5" exitCode=0
Oct 30 21:21:20.103400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.103348    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5" event={"ID":"4074b6d4-ca47-47ba-9677-0e0b9ba2099d","Type":"ContainerDied","Data":"6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5"}
Oct 30 21:21:20.103400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.103369    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"
Oct 30 21:21:20.103686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.103400    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5" event={"ID":"4074b6d4-ca47-47ba-9677-0e0b9ba2099d","Type":"ContainerDied","Data":"011ccb1e857932fcfe4fcb3ac7dd6b54e79e2b7a62d23338ac6d956c018e7f26"}
Oct 30 21:21:20.103686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.103442    4633 scope.go:122] "RemoveContainer" containerID="6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5"
Oct 30 21:21:20.138758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.135080    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsb4p\" (UniqueName: \"kubernetes.io/projected/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-kube-api-access-qsb4p\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:20.138758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.135121    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:20.138758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.135138    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:20.138758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.135152    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4074b6d4-ca47-47ba-9677-0e0b9ba2099d-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:20.142874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.142834    4633 scope.go:122] "RemoveContainer" containerID="6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5"
Oct 30 21:21:20.143292 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:20.143271    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5\": container with ID starting with 6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5 not found: ID does not exist" containerID="6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5"
Oct 30 21:21:20.143340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.143302    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5"} err="failed to get container status \"6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5\": rpc error: code = NotFound desc = could not find container \"6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5\": container with ID starting with 6f76b9c06139a3cc06d5cb67edd7babafe7f6b633f4ff5b8393caf08227bccf5 not found: ID does not exist"
Oct 30 21:21:20.164821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.164785    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"]
Oct 30 21:21:20.173970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:20.173945    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ac6bf-6b786b77c9-pf2j5"]
Oct 30 21:21:21.792893 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:21.792862    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4074b6d4-ca47-47ba-9677-0e0b9ba2099d" path="/var/lib/kubelet/pods/4074b6d4-ca47-47ba-9677-0e0b9ba2099d/volumes"
Oct 30 21:21:22.110196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.110113    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"]
Oct 30 21:21:22.111216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111192    4633 cpu_manager.go:414] "containerMap: removing container" podUID="436e8e3d-1cd0-4e50-91f1-60004acfd7f2" containerName="payload-processing"
Oct 30 21:21:22.111333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111220    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436e8e3d-1cd0-4e50-91f1-60004acfd7f2" containerName="payload-processing"
Oct 30 21:21:22.111333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111236    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1d520039-aae0-4404-b51f-4e1d0477692a" containerName="payload-pre-processing"
Oct 30 21:21:22.111333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111243    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d520039-aae0-4404-b51f-4e1d0477692a" containerName="payload-pre-processing"
Oct 30 21:21:22.111333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111285    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy"
Oct 30 21:21:22.111333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111294    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy"
Oct 30 21:21:22.111577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111344    4633 cpu_manager.go:414] "containerMap: removing container" podUID="4074b6d4-ca47-47ba-9677-0e0b9ba2099d" containerName="maas-api"
Oct 30 21:21:22.111577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111351    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4074b6d4-ca47-47ba-9677-0e0b9ba2099d" containerName="maas-api"
Oct 30 21:21:22.111577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111490    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy"
Oct 30 21:21:22.111577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111509    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4074b6d4-ca47-47ba-9677-0e0b9ba2099d" containerName="maas-api"
Oct 30 21:21:22.111577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111519    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy"
Oct 30 21:21:22.111577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111529    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="436e8e3d-1cd0-4e50-91f1-60004acfd7f2" containerName="payload-processing"
Oct 30 21:21:22.111577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.111539    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d520039-aae0-4404-b51f-4e1d0477692a" containerName="payload-pre-processing"
Oct 30 21:21:22.161600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.161565    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.162126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.161885    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"]
Oct 30 21:21:22.165981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.165158    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d87b5f-openshift-default-dockercfg-2rdjp\""
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.278738    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-token\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.278827    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjnn\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-kube-api-access-mwjnn\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.278973    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-socket\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.279020    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-envoy\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.279050    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-data\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.279082    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-podinfo\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.279141    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ce198d-a5bb-4115-bffd-206c9faf4629-istiod-ca-cert\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.279288    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-credential-socket\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.283743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.279334    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-certs\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.381697    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-credential-socket\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.381758    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-certs\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382156    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-certs\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382210    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-credential-socket\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382263    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-token\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382324    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjnn\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-kube-api-access-mwjnn\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382465    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-socket\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382516    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-envoy\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382550    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-data\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382584    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-podinfo\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.382644    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ce198d-a5bb-4115-bffd-206c9faf4629-istiod-ca-cert\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.388733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.383507    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ce198d-a5bb-4115-bffd-206c9faf4629-istiod-ca-cert\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.410961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.404357    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-socket\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.410961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.407005    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-data\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.436936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.412437    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-token\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.436936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.435138    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-envoy\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.446717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.437326    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-podinfo\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.496709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.489342    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjnn\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-kube-api-access-mwjnn\") pod \"e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") " pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.534242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.509165    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:22.994691 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:21:22.994648    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ce198d_a5bb_4115_bffd_206c9faf4629.slice/crio-faf23fe0d8f6689f2251e9b2e6f162ab26a9df7ba6462e0ceb15b41c37b1ca3a WatchSource:0}: Error finding container faf23fe0d8f6689f2251e9b2e6f162ab26a9df7ba6462e0ceb15b41c37b1ca3a: Status 404 returned error can't find the container with id faf23fe0d8f6689f2251e9b2e6f162ab26a9df7ba6462e0ceb15b41c37b1ca3a
Oct 30 21:21:22.995401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.995377    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"]
Oct 30 21:21:22.996967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.996936    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:22.997032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.996999    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:22.997070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:22.997030    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:23.178326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:23.178286    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" event={"ID":"46ce198d-a5bb-4115-bffd-206c9faf4629","Type":"ContainerStarted","Data":"47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11"}
Oct 30 21:21:23.178529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:23.178366    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" event={"ID":"46ce198d-a5bb-4115-bffd-206c9faf4629","Type":"ContainerStarted","Data":"faf23fe0d8f6689f2251e9b2e6f162ab26a9df7ba6462e0ceb15b41c37b1ca3a"}
Oct 30 21:21:23.203812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:23.203746    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podStartSLOduration=1.203729574 podStartE2EDuration="1.203729574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:23.198476397 +0000 UTC m=+1191.548203810" watchObservedRunningTime="2026-10-30 21:21:23.203729574 +0000 UTC m=+1191.553456970"
Oct 30 21:21:23.510062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:23.510011    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:24.045896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.045864    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"]
Oct 30 21:21:24.050798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.050774    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1383e8ec-v259q"]
Oct 30 21:21:24.544878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.544844    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:24.963231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.961002    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"]
Oct 30 21:21:24.963231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.962117    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy"
Oct 30 21:21:24.963231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.962139    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd236d7-7f72-49a1-850c-642a23a4ef34" containerName="istio-proxy"
Oct 30 21:21:24.967136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.967025    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"]
Oct 30 21:21:24.967316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.967195    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:24.974073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:24.973690    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-09eb50-openshift-default-dockercfg-929p8\""
Oct 30 21:21:25.159962 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:25.159925    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:21:25.179869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.162821    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-socket\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.180239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.180214    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b4g9\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-kube-api-access-2b4g9\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.180466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.180446    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-credential-socket\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.180625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.180608    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-certs\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.198743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.189750    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-data\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.198743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.189821    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-token\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.198743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.189849    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-podinfo\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.198743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.189914    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istiod-ca-cert\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.198743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.190101    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-envoy\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.291819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.291713    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-socket\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.303118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.303085    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b4g9\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-kube-api-access-2b4g9\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.303855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.303827    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-credential-socket\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.334896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.334863    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-certs\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.335533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.335506    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-data\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.345754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.336093    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-token\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.347692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.347643    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-podinfo\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.348494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.348474    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istiod-ca-cert\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.364294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.364260    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-envoy\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.364638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.335434    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-certs\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.364790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.292434    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-socket\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.365416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.349297    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istiod-ca-cert\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.365550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.361884    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b4g9\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-kube-api-access-2b4g9\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.365649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.336016    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-data\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.366150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.304419    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-credential-socket\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.369308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.369285    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:25.396314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.396227    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-podinfo\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.397087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.396782    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:21:25.397819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.397796    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-token\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.398229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.398190    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-envoy\") pod \"e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") " pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:25.595275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:25.595187    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:26.069536 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.069496    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a57313-86cf-4b26-a3a6-4e3d660c674d" path="/var/lib/kubelet/pods/63a57313-86cf-4b26-a3a6-4e3d660c674d/volumes"
Oct 30 21:21:26.137779 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:21:26.137744    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730d60c8_aa12_4e39_9fb7_c43aeb8aa656.slice/crio-df4eeb2c0cd7d6cfb1073b10e3da767a29144ba9c62c33007cc6827aa86eeb25 WatchSource:0}: Error finding container df4eeb2c0cd7d6cfb1073b10e3da767a29144ba9c62c33007cc6827aa86eeb25: Status 404 returned error can't find the container with id df4eeb2c0cd7d6cfb1073b10e3da767a29144ba9c62c33007cc6827aa86eeb25
Oct 30 21:21:26.138597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.138572    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"]
Oct 30 21:21:26.141031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.140999    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:26.141137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.141082    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:26.141137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.141112    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:26.379820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.379710    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" event={"ID":"730d60c8-aa12-4e39-9fb7-c43aeb8aa656","Type":"ContainerStarted","Data":"54a06d4426560ba0962609f24997fb522a0becf6e03c89a1a341e35be1d90084"}
Oct 30 21:21:26.379820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.379769    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" event={"ID":"730d60c8-aa12-4e39-9fb7-c43aeb8aa656","Type":"ContainerStarted","Data":"df4eeb2c0cd7d6cfb1073b10e3da767a29144ba9c62c33007cc6827aa86eeb25"}
Oct 30 21:21:26.404295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.404245    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podStartSLOduration=2.404226622 podStartE2EDuration="2.404226622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:26.401141548 +0000 UTC m=+1194.750868949" watchObservedRunningTime="2026-10-30 21:21:26.404226622 +0000 UTC m=+1194.753954018"
Oct 30 21:21:26.625693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:26.620448    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:27.602830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:27.602798    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:27.796135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:27.796096    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"]
Oct 30 21:21:27.796325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:27.796217    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"]
Oct 30 21:21:27.796469 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:27.796439    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy" containerID="cri-o://f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7" gracePeriod=30
Oct 30 21:21:27.797154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:27.797014    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy" containerID="cri-o://8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029" gracePeriod=30
Oct 30 21:21:28.415339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:28.415295    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:28.416306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:28.416284    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:21:30.038013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.037977    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"]
Oct 30 21:21:30.054967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.054942    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"]
Oct 30 21:21:30.055119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.055044    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:30.076181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.076153    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09eb50\""
Oct 30 21:21:30.076439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.076422    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09eb50-dockercfg-lpr2x\""
Oct 30 21:21:30.173695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.162156    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1410ef-3273-4786-8af4-2db7aa84a312-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf\" (UID: \"3b1410ef-3273-4786-8af4-2db7aa84a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:30.173695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.162237    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dz2\" (UniqueName: \"kubernetes.io/projected/3b1410ef-3273-4786-8af4-2db7aa84a312-kube-api-access-n4dz2\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf\" (UID: \"3b1410ef-3273-4786-8af4-2db7aa84a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:30.369440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.367453    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1410ef-3273-4786-8af4-2db7aa84a312-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf\" (UID: \"3b1410ef-3273-4786-8af4-2db7aa84a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:30.369440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.367529    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dz2\" (UniqueName: \"kubernetes.io/projected/3b1410ef-3273-4786-8af4-2db7aa84a312-kube-api-access-n4dz2\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf\" (UID: \"3b1410ef-3273-4786-8af4-2db7aa84a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:30.369440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.368494    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1410ef-3273-4786-8af4-2db7aa84a312-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf\" (UID: \"3b1410ef-3273-4786-8af4-2db7aa84a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:30.418285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.418176    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dz2\" (UniqueName: \"kubernetes.io/projected/3b1410ef-3273-4786-8af4-2db7aa84a312-kube-api-access-n4dz2\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf\" (UID: \"3b1410ef-3273-4786-8af4-2db7aa84a312\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:30.419017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.418973    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:30.769995 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:21:30.769966    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1410ef_3273_4786_8af4_2db7aa84a312.slice/crio-bafb613651979985e4dc69ed412cb6bbe1175e7a265b27d41022b1e68bd4397b WatchSource:0}: Error finding container bafb613651979985e4dc69ed412cb6bbe1175e7a265b27d41022b1e68bd4397b: Status 404 returned error can't find the container with id bafb613651979985e4dc69ed412cb6bbe1175e7a265b27d41022b1e68bd4397b
Oct 30 21:21:30.774227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:30.774091    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"]
Oct 30 21:21:31.451319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:31.451286    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf" event={"ID":"3b1410ef-3273-4786-8af4-2db7aa84a312","Type":"ContainerStarted","Data":"ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807"}
Oct 30 21:21:31.451823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:31.451341    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf" event={"ID":"3b1410ef-3273-4786-8af4-2db7aa84a312","Type":"ContainerStarted","Data":"bafb613651979985e4dc69ed412cb6bbe1175e7a265b27d41022b1e68bd4397b"}
Oct 30 21:21:31.451823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:31.451386    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:31.470952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:31.470896    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf" podStartSLOduration=1.4708782710000001 podStartE2EDuration="1.470878271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:31.467625501 +0000 UTC m=+1199.817352896" watchObservedRunningTime="2026-10-30 21:21:31.470878271 +0000 UTC m=+1199.820605668"
Oct 30 21:21:32.008896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.008869    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:21:32.028541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.028466    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:21:32.038532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.038506    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:21:32.055762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.055739    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:21:32.461680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.461634    4633 generic.go:356] "Generic (PLEG): container finished" podID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerID="b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b" exitCode=0
Oct 30 21:21:32.462205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.461728    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" event={"ID":"f01d8427-ce69-4098-b6c9-cf2eb7530091","Type":"ContainerDied","Data":"b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b"}
Oct 30 21:21:32.462532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.462514    4633 scope.go:122] "RemoveContainer" containerID="b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b"
Oct 30 21:21:32.464111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.464080    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:32.464197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.464152    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:32.464197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:32.464186    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:33.133850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.133821    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:21:33.174286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.174261    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-token\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-token\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.174427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.174318    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-podinfo\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.174427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.174351    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-data\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.174427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.174409    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-envoy\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.174529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.174514    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-kube-api-access-9nlfb\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-kube-api-access-9nlfb\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.174811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.174790    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-socket\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.175269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.174876    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-data" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.175396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.175153    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-socket" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.175540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.175256    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-credential-socket\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.175634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.175535    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-credential-socket" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.175819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.175762    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-certs\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.175910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.175847    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-certs" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.176034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.176016    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istiod-ca-cert\") pod \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\" (UID: \"8fc53139-a140-4d72-a29f-fda75ddb7d5e\") "
Oct 30 21:21:33.176409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.176386    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istiod-ca-cert" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:33.176798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.176779    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.176885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.176803    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.176885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.176817    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.176885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.176833    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.176885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.176846    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.177780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.177754    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-token" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:33.178479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.178452    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-envoy" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.179047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.179025    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-podinfo" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:21:33.179175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.179159    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-kube-api-access-9nlfb" pod "8fc53139-a140-4d72-a29f-fda75ddb7d5e" (UID: "8fc53139-a140-4d72-a29f-fda75ddb7d5e"). InnerVolumeSpecName "kube-api-access-9nlfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:33.215510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.215487    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:21:33.277747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.277709    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-podinfo\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.277924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.277761    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-socket\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.277924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.277812    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-kube-api-access-hdd7g\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-kube-api-access-hdd7g\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.277924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.277846    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-credential-socket\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.278133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.278065    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-envoy\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.278234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.278196    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-socket" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.278234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.278216    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-certs\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.278485 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.278459    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-certs" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.278589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.278570    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-token\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-token\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.278655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.278585    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-credential-socket" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.278970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.278950    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-data\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.279076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279010    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec77c66-fcd5-43c0-aa30-55d522a0a157-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec77c66-fcd5-43c0-aa30-55d522a0a157-istiod-ca-cert\") pod \"bec77c66-fcd5-43c0-aa30-55d522a0a157\" (UID: \"bec77c66-fcd5-43c0-aa30-55d522a0a157\") "
Oct 30 21:21:33.279499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279461    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec77c66-fcd5-43c0-aa30-55d522a0a157-istiod-ca-cert" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:33.279902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279877    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.280021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279908    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.280021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279923    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.280021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279938    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.280021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279952    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fc53139-a140-4d72-a29f-fda75ddb7d5e-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.280021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279966    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.280021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279982    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nlfb\" (UniqueName: \"kubernetes.io/projected/8fc53139-a140-4d72-a29f-fda75ddb7d5e-kube-api-access-9nlfb\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.280021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.279999    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec77c66-fcd5-43c0-aa30-55d522a0a157-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.280425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.280172    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-data" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.280782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.280756    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-podinfo" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:21:33.280954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.280935    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-envoy" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.281742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.281718    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-token" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:33.282173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.282148    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-kube-api-access-hdd7g" pod "bec77c66-fcd5-43c0-aa30-55d522a0a157" (UID: "bec77c66-fcd5-43c0-aa30-55d522a0a157"). InnerVolumeSpecName "kube-api-access-hdd7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:33.395228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.383483    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdd7g\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-kube-api-access-hdd7g\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.395228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.383522    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.395228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.383538    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.395228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.383551    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.395228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.383563    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec77c66-fcd5-43c0-aa30-55d522a0a157-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.476881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.476842    4633 generic.go:356] "Generic (PLEG): container finished" podID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerID="f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7" exitCode=0
Oct 30 21:21:33.477303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.476890    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"
Oct 30 21:21:33.477303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.476913    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" event={"ID":"8fc53139-a140-4d72-a29f-fda75ddb7d5e","Type":"ContainerDied","Data":"f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7"}
Oct 30 21:21:33.477303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.476965    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg" event={"ID":"8fc53139-a140-4d72-a29f-fda75ddb7d5e","Type":"ContainerDied","Data":"ad6b61902f940578aa3a762050c4f862222926585781aa08bd723f02b898f05d"}
Oct 30 21:21:33.477303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.476986    4633 scope.go:122] "RemoveContainer" containerID="f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7"
Oct 30 21:21:33.478621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.478592    4633 generic.go:356] "Generic (PLEG): container finished" podID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerID="8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029" exitCode=0
Oct 30 21:21:33.478846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.478767    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"
Oct 30 21:21:33.479377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.479356    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" event={"ID":"bec77c66-fcd5-43c0-aa30-55d522a0a157","Type":"ContainerDied","Data":"8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029"}
Oct 30 21:21:33.479451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.479406    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6" event={"ID":"bec77c66-fcd5-43c0-aa30-55d522a0a157","Type":"ContainerDied","Data":"6982e6ac1b76ea27eec58d9e4b8214a2e2ed051942946ba3d0db7864f6f6456d"}
Oct 30 21:21:33.481885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.481858    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" event={"ID":"f01d8427-ce69-4098-b6c9-cf2eb7530091","Type":"ContainerStarted","Data":"1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece"}
Oct 30 21:21:33.491226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.491209    4633 scope.go:122] "RemoveContainer" containerID="f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7"
Oct 30 21:21:33.491525 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:33.491508    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7\": container with ID starting with f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7 not found: ID does not exist" containerID="f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7"
Oct 30 21:21:33.491578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.491533    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7"} err="failed to get container status \"f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7\": rpc error: code = NotFound desc = could not find container \"f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7\": container with ID starting with f821a14da91197beeef52eb7593bb0e0b149c7092d1b85f3dcc20d352b0dbda7 not found: ID does not exist"
Oct 30 21:21:33.491578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.491549    4633 scope.go:122] "RemoveContainer" containerID="8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029"
Oct 30 21:21:33.502437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.502420    4633 scope.go:122] "RemoveContainer" containerID="8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029"
Oct 30 21:21:33.502769 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:33.502747    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029\": container with ID starting with 8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029 not found: ID does not exist" containerID="8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029"
Oct 30 21:21:33.502848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.502780    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029"} err="failed to get container status \"8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029\": rpc error: code = NotFound desc = could not find container \"8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029\": container with ID starting with 8f13816b3f65f1502c2fc88f4d3681f8e0c02222d8bd6bf6b79144af9af2d029 not found: ID does not exist"
Oct 30 21:21:33.541694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.541274    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"]
Oct 30 21:21:33.545383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.545356    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-2pmp6"]
Oct 30 21:21:33.550965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.550944    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"]
Oct 30 21:21:33.554974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.554951    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-6nmjg"]
Oct 30 21:21:33.611622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.611596    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:21:33.612597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.612574    4633 patch_prober.go:29] interesting pod/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:21:33.612780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.612619    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:21:33.690458 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:33.690418    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:21:33.791703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.791628    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" path="/var/lib/kubelet/pods/8fc53139-a140-4d72-a29f-fda75ddb7d5e/volumes"
Oct 30 21:21:33.792321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:33.792296    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" path="/var/lib/kubelet/pods/bec77c66-fcd5-43c0-aa30-55d522a0a157/volumes"
Oct 30 21:21:34.054061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:34.053963    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"]
Oct 30 21:21:34.058485 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:34.058449    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fd631b6-9zkrb"]
Oct 30 21:21:34.611506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:34.611471    4633 patch_prober.go:29] interesting pod/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:21:34.611969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:34.611534    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:21:35.611251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.611218    4633 patch_prober.go:29] interesting pod/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:21:35.611430 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.611278    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:21:35.657880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.654118    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"]
Oct 30 21:21:35.657880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.655122    4633 cpu_manager.go:414] "containerMap: removing container" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy"
Oct 30 21:21:35.657880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.655143    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy"
Oct 30 21:21:35.657880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.655181    4633 cpu_manager.go:414] "containerMap: removing container" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy"
Oct 30 21:21:35.657880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.655190    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy"
Oct 30 21:21:35.657880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.655412    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec77c66-fcd5-43c0-aa30-55d522a0a157" containerName="istio-proxy"
Oct 30 21:21:35.657880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.655435    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc53139-a140-4d72-a29f-fda75ddb7d5e" containerName="istio-proxy"
Oct 30 21:21:35.661113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.661091    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.677841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.677818    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d87b5f-serving-cert\""
Oct 30 21:21:35.693848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.693810    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"]
Oct 30 21:21:35.758204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.757983    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.785713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.783588    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958f15d6-c86f-41f5-9f21-d7774f585966-openshift-service-ca\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.785713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.783947    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-metrics-certs\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.785713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.784070    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxsmr\" (UniqueName: \"kubernetes.io/projected/958f15d6-c86f-41f5-9f21-d7774f585966-kube-api-access-rxsmr\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.893794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.893695    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-metrics-certs\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.894127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.894102    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxsmr\" (UniqueName: \"kubernetes.io/projected/958f15d6-c86f-41f5-9f21-d7774f585966-kube-api-access-rxsmr\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.897698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.894333    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.897698 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:35.894627    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-serving-cert: secret "maas-api-e2e-api-b-d87b5f-serving-cert" not found
Oct 30 21:21:35.897698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.895071    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958f15d6-c86f-41f5-9f21-d7774f585966-openshift-service-ca\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.897698 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:35.895097    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls podName:958f15d6-c86f-41f5-9f21-d7774f585966 nodeName:}" failed. No retries permitted until 2026-10-30 21:21:36.395072465 +0000 UTC m=+1204.744799852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls") pod "maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q" (UID: "958f15d6-c86f-41f5-9f21-d7774f585966") : secret "maas-api-e2e-api-b-d87b5f-serving-cert" not found
Oct 30 21:21:35.897698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.895939    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958f15d6-c86f-41f5-9f21-d7774f585966-openshift-service-ca\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.898768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.898742    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-metrics-certs\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.904254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.904234    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxsmr\" (UniqueName: \"kubernetes.io/projected/958f15d6-c86f-41f5-9f21-d7774f585966-kube-api-access-rxsmr\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:35.908096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:35.908070    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830e6413-8bf5-46b8-a643-08f403366a81" path="/var/lib/kubelet/pods/830e6413-8bf5-46b8-a643-08f403366a81/volumes"
Oct 30 21:21:36.424696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:36.413235    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:36.424696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:36.418458    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls\") pod \"maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:36.589634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:36.589592    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:36.612214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:36.612177    4633 patch_prober.go:29] interesting pod/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:21:36.612321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:36.612228    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:21:36.962546 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:21:36.962519    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958f15d6_c86f_41f5_9f21_d7774f585966.slice/crio-dca122e5925d096f3da8e9bb6a20aa04129156fe0e3ffe105bf3098888b7445c WatchSource:0}: Error finding container dca122e5925d096f3da8e9bb6a20aa04129156fe0e3ffe105bf3098888b7445c: Status 404 returned error can't find the container with id dca122e5925d096f3da8e9bb6a20aa04129156fe0e3ffe105bf3098888b7445c
Oct 30 21:21:36.962898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:36.962729    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"]
Oct 30 21:21:37.547147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:37.547114    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q" event={"ID":"958f15d6-c86f-41f5-9f21-d7774f585966","Type":"ContainerStarted","Data":"dca122e5925d096f3da8e9bb6a20aa04129156fe0e3ffe105bf3098888b7445c"}
Oct 30 21:21:37.611722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:37.611685    4633 patch_prober.go:29] interesting pod/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:21:37.611914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:37.611754    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:21:38.568248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:38.563663    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q" event={"ID":"958f15d6-c86f-41f5-9f21-d7774f585966","Type":"ContainerStarted","Data":"88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78"}
Oct 30 21:21:38.568248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:38.565044    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:38.601883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:38.601830    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q" podStartSLOduration=3.070580215 podStartE2EDuration="3.60181632s" totalImagesPullingTime="531.236105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:38.598279949 +0000 UTC m=+1206.948007346" watchObservedRunningTime="2026-10-30 21:21:38.60181632 +0000 UTC m=+1206.951543716"
Oct 30 21:21:38.611302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:38.611265    4633 patch_prober.go:29] interesting pod/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:21:38.611481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:38.611325    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 30 21:21:39.633877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:39.633847    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:21:39.636037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:39.636009    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:21:39.637035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:39.637017    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:21:42.466090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:42.466008    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:43.793089 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:43.793060    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:21:44.269379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.269344    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"]
Oct 30 21:21:44.269884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.269831    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn" podUID="ec016bee-4fb5-4681-9ec6-7b9633d42af6" containerName="maas-api" containerID="cri-o://2e8126f42a7888af9af36c48a5a833f4484b6f311453f3b3f2a8a3ff13095fca" gracePeriod=30
Oct 30 21:21:44.352403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.352369    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"]
Oct 30 21:21:44.352706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.352660    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp" podUID="399a9845-1ede-4514-9177-049998f244aa" containerName="payload-processing" containerID="cri-o://2bbac29f8bcbd7faac1b189a95d28b37ab43e437327ef8bb65095f6a5b5802b5" gracePeriod=30
Oct 30 21:21:44.846439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.846245    4633 generic.go:356] "Generic (PLEG): container finished" podID="ec016bee-4fb5-4681-9ec6-7b9633d42af6" containerID="2e8126f42a7888af9af36c48a5a833f4484b6f311453f3b3f2a8a3ff13095fca" exitCode=0
Oct 30 21:21:44.846439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.846359    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn" event={"ID":"ec016bee-4fb5-4681-9ec6-7b9633d42af6","Type":"ContainerDied","Data":"2e8126f42a7888af9af36c48a5a833f4484b6f311453f3b3f2a8a3ff13095fca"}
Oct 30 21:21:44.849750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.849407    4633 generic.go:356] "Generic (PLEG): container finished" podID="399a9845-1ede-4514-9177-049998f244aa" containerID="2bbac29f8bcbd7faac1b189a95d28b37ab43e437327ef8bb65095f6a5b5802b5" exitCode=0
Oct 30 21:21:44.849750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.849644    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp" event={"ID":"399a9845-1ede-4514-9177-049998f244aa","Type":"ContainerDied","Data":"2bbac29f8bcbd7faac1b189a95d28b37ab43e437327ef8bb65095f6a5b5802b5"}
Oct 30 21:21:44.891764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.891731    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:21:44.907460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.907428    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls\") pod \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") "
Oct 30 21:21:44.907607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.907541    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec016bee-4fb5-4681-9ec6-7b9633d42af6-kube-api-access-2m4r6\" (UniqueName: \"kubernetes.io/projected/ec016bee-4fb5-4681-9ec6-7b9633d42af6-kube-api-access-2m4r6\") pod \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") "
Oct 30 21:21:44.907607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.907566    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-metrics-certs\") pod \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") "
Oct 30 21:21:44.907701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.907633    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec016bee-4fb5-4681-9ec6-7b9633d42af6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec016bee-4fb5-4681-9ec6-7b9633d42af6-openshift-service-ca\") pod \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\" (UID: \"ec016bee-4fb5-4681-9ec6-7b9633d42af6\") "
Oct 30 21:21:44.908146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.908115    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec016bee-4fb5-4681-9ec6-7b9633d42af6-openshift-service-ca" pod "ec016bee-4fb5-4681-9ec6-7b9633d42af6" (UID: "ec016bee-4fb5-4681-9ec6-7b9633d42af6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:44.908514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.908491    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec016bee-4fb5-4681-9ec6-7b9633d42af6-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:44.909886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.909860    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls" pod "ec016bee-4fb5-4681-9ec6-7b9633d42af6" (UID: "ec016bee-4fb5-4681-9ec6-7b9633d42af6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:44.910788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.910758    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec016bee-4fb5-4681-9ec6-7b9633d42af6-kube-api-access-2m4r6" pod "ec016bee-4fb5-4681-9ec6-7b9633d42af6" (UID: "ec016bee-4fb5-4681-9ec6-7b9633d42af6"). InnerVolumeSpecName "kube-api-access-2m4r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:44.911005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.910985    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-metrics-certs" pod "ec016bee-4fb5-4681-9ec6-7b9633d42af6" (UID: "ec016bee-4fb5-4681-9ec6-7b9633d42af6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:44.978315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:44.978286    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:21:45.009794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.009759    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399a9845-1ede-4514-9177-049998f244aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399a9845-1ede-4514-9177-049998f244aa-plugins-config-volume\") pod \"399a9845-1ede-4514-9177-049998f244aa\" (UID: \"399a9845-1ede-4514-9177-049998f244aa\") "
Oct 30 21:21:45.010034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.009894    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399a9845-1ede-4514-9177-049998f244aa-kube-api-access-4zzz2\" (UniqueName: \"kubernetes.io/projected/399a9845-1ede-4514-9177-049998f244aa-kube-api-access-4zzz2\") pod \"399a9845-1ede-4514-9177-049998f244aa\" (UID: \"399a9845-1ede-4514-9177-049998f244aa\") "
Oct 30 21:21:45.010493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.010466    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399a9845-1ede-4514-9177-049998f244aa-plugins-config-volume" pod "399a9845-1ede-4514-9177-049998f244aa" (UID: "399a9845-1ede-4514-9177-049998f244aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:45.010626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.010608    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/399a9845-1ede-4514-9177-049998f244aa-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:45.010714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.010634    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:45.010714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.010651    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4r6\" (UniqueName: \"kubernetes.io/projected/ec016bee-4fb5-4681-9ec6-7b9633d42af6-kube-api-access-2m4r6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:45.010714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.010683    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec016bee-4fb5-4681-9ec6-7b9633d42af6-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:45.013433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.013406    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399a9845-1ede-4514-9177-049998f244aa-kube-api-access-4zzz2" pod "399a9845-1ede-4514-9177-049998f244aa" (UID: "399a9845-1ede-4514-9177-049998f244aa"). InnerVolumeSpecName "kube-api-access-4zzz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:45.112684 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.112557    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zzz2\" (UniqueName: \"kubernetes.io/projected/399a9845-1ede-4514-9177-049998f244aa-kube-api-access-4zzz2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:45.644625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.644590    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:21:45.860196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.860173    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"
Oct 30 21:21:45.860196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.860183    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn" event={"ID":"ec016bee-4fb5-4681-9ec6-7b9633d42af6","Type":"ContainerDied","Data":"efa8bfa4d9937552dfc33837775bebc70fe5bd9f8744532500d08980bdd5f924"}
Oct 30 21:21:45.860609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.860247    4633 scope.go:122] "RemoveContainer" containerID="2e8126f42a7888af9af36c48a5a833f4484b6f311453f3b3f2a8a3ff13095fca"
Oct 30 21:21:45.861921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.861889    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp" event={"ID":"399a9845-1ede-4514-9177-049998f244aa","Type":"ContainerDied","Data":"278af5b159391fcaf6faa5bbbf521da4d79241a5be7bb530ac6af694ec25ccc4"}
Oct 30 21:21:45.862019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.861927    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"
Oct 30 21:21:45.871776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.871750    4633 scope.go:122] "RemoveContainer" containerID="2bbac29f8bcbd7faac1b189a95d28b37ab43e437327ef8bb65095f6a5b5802b5"
Oct 30 21:21:45.890165 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.890139    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"]
Oct 30 21:21:45.894331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.894300    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60c26c-545fbb8b64-8g8mp"]
Oct 30 21:21:45.909559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.909526    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"]
Oct 30 21:21:45.917460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:45.917432    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60c26c-fdbbbb96f-s6lmn"]
Oct 30 21:21:46.793511 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:46.793466    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:21:46.881155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.881120    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"]
Oct 30 21:21:46.882236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.882214    4633 cpu_manager.go:414] "containerMap: removing container" podUID="399a9845-1ede-4514-9177-049998f244aa" containerName="payload-processing"
Oct 30 21:21:46.882345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.882239    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399a9845-1ede-4514-9177-049998f244aa" containerName="payload-processing"
Oct 30 21:21:46.882345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.882262    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ec016bee-4fb5-4681-9ec6-7b9633d42af6" containerName="maas-api"
Oct 30 21:21:46.882345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.882267    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec016bee-4fb5-4681-9ec6-7b9633d42af6" containerName="maas-api"
Oct 30 21:21:46.882508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.882418    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="399a9845-1ede-4514-9177-049998f244aa" containerName="payload-processing"
Oct 30 21:21:46.882508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.882428    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec016bee-4fb5-4681-9ec6-7b9633d42af6" containerName="maas-api"
Oct 30 21:21:46.889242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.889209    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.893622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.893589    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"]
Oct 30 21:21:46.922942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.922905    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4c6998-openshift-default-dockercfg-dpkjz\""
Oct 30 21:21:46.976797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.976756    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-certs\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.977076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.977048    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-podinfo\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.977278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.977255    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvwzl\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-kube-api-access-nvwzl\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.977480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.977434    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-data\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.977480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.977472    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istiod-ca-cert\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.977652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.977505    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-token\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.977652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.977598    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-credential-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.977771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.977712    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-envoy\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:46.977892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:46.977877    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079396    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvwzl\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-kube-api-access-nvwzl\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079451    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-data\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079471    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istiod-ca-cert\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079492    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-token\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079578    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-credential-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079661    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-envoy\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079853    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079931    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-certs\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.079950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079935    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-data\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.080170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.079958    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-podinfo\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.080374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.080351    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-credential-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.080509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.080491    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-certs\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.080582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.080527    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.080992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.080970    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istiod-ca-cert\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.082906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.082879    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-podinfo\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.083025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.082888    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-envoy\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.087909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.087875    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-token\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.088005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.087877    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvwzl\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-kube-api-access-nvwzl\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.207622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.207578    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:47.498290 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.498259    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"]
Oct 30 21:21:47.501809 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:21:47.501761    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c1fb3c_70ac_41da_b1e3_86e307fc9369.slice/crio-1ab25d33b2499f1a49a05a63a5c0e2f4a954057c028ddfe8df31fe005d279bcb WatchSource:0}: Error finding container 1ab25d33b2499f1a49a05a63a5c0e2f4a954057c028ddfe8df31fe005d279bcb: Status 404 returned error can't find the container with id 1ab25d33b2499f1a49a05a63a5c0e2f4a954057c028ddfe8df31fe005d279bcb
Oct 30 21:21:47.504835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.504796    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:47.504952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.504897    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:47.504952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.504946    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:47.794521 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.794437    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399a9845-1ede-4514-9177-049998f244aa" path="/var/lib/kubelet/pods/399a9845-1ede-4514-9177-049998f244aa/volumes"
Oct 30 21:21:47.795031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.795011    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec016bee-4fb5-4681-9ec6-7b9633d42af6" path="/var/lib/kubelet/pods/ec016bee-4fb5-4681-9ec6-7b9633d42af6/volumes"
Oct 30 21:21:47.901722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.901689    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" event={"ID":"70c1fb3c-70ac-41da-b1e3-86e307fc9369","Type":"ContainerStarted","Data":"9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef"}
Oct 30 21:21:47.902139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.901741    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" event={"ID":"70c1fb3c-70ac-41da-b1e3-86e307fc9369","Type":"ContainerStarted","Data":"1ab25d33b2499f1a49a05a63a5c0e2f4a954057c028ddfe8df31fe005d279bcb"}
Oct 30 21:21:47.924964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:47.924917    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podStartSLOduration=1.92489951 podStartE2EDuration="1.92489951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:47.920817004 +0000 UTC m=+1216.270544416" watchObservedRunningTime="2026-10-30 21:21:47.92489951 +0000 UTC m=+1216.274626906"
Oct 30 21:21:48.208283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:48.208249    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:48.210118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:48.210089    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 21:21:48.210262 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:48.210136    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 21:21:49.211545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.211074    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 21:21:49.211545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.211137    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 21:21:49.609750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.607895    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"]
Oct 30 21:21:49.651942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.651908    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"]
Oct 30 21:21:49.652233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.652208    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.733005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.732752    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-6563b6-openshift-default-dockercfg-6rgpg\""
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.816645    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istiod-ca-cert\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.816808    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-envoy\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.816873    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.816912    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7m5\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-kube-api-access-qj7m5\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.816975    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-podinfo\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.817106    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-token\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.817141    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-credential-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.817214    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-data\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.837828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.817275    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-certs\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.921423    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-envoy\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.921502    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.921538    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7m5\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-kube-api-access-qj7m5\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.921601    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-podinfo\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.921791    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-token\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.921844    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-credential-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.921922    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-data\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.922001    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-certs\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.926696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.922066    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istiod-ca-cert\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.949313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.938567    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-certs\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.949313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.942268    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-data\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.949907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.949844    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.950636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.950587    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istiod-ca-cert\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.950993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.950973    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-credential-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:49.991702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:49.990392    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7m5\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-kube-api-access-qj7m5\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:50.014700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.013290    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-token\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:50.037696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.021119    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-envoy\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:50.052304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.052265    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-podinfo\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:50.213460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.213366    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 21:21:50.214074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.214040    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 21:21:50.270175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.270139    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:50.595617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.595589    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"]
Oct 30 21:21:50.595758 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:21:50.595652    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020842ff_b53f_47ce_a40f_72e4e56f2c2c.slice/crio-f7aaf1f5295aec4446a67ebc4220c35939953881664a74094a88b2e4a59d9a9b WatchSource:0}: Error finding container f7aaf1f5295aec4446a67ebc4220c35939953881664a74094a88b2e4a59d9a9b: Status 404 returned error can't find the container with id f7aaf1f5295aec4446a67ebc4220c35939953881664a74094a88b2e4a59d9a9b
Oct 30 21:21:50.598569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.598535    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:50.598649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.598616    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:50.598649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:50.598645    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:51.127406 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:51.127364    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" event={"ID":"020842ff-b53f-47ce-a40f-72e4e56f2c2c","Type":"ContainerStarted","Data":"62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78"}
Oct 30 21:21:51.142791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:51.142755    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" event={"ID":"020842ff-b53f-47ce-a40f-72e4e56f2c2c","Type":"ContainerStarted","Data":"f7aaf1f5295aec4446a67ebc4220c35939953881664a74094a88b2e4a59d9a9b"}
Oct 30 21:21:51.212179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:51.211743    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 21:21:51.212179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:51.211808    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 21:21:51.271700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:51.271181    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:51.272726 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:51.272701    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:21:51.272846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:51.272750    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:21:52.208841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:52.208810    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 21:21:52.208989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:52.208864    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 21:21:52.289390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:52.288911    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:52.321703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:52.320196    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podStartSLOduration=3.320177231 podStartE2EDuration="3.320177231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:51.213701785 +0000 UTC m=+1219.563429204" watchObservedRunningTime="2026-10-30 21:21:52.320177231 +0000 UTC m=+1220.669904628"
Oct 30 21:21:53.199175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:53.199100    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:53.209767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:53.204982    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:21:53.209767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:53.208110    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 21:21:53.209767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:53.208161    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 21:21:53.905995 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:53.905962    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:21:53.953255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:53.953217    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"]
Oct 30 21:21:54.208642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:54.208563    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 21:21:54.208642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:54.208618    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 21:21:55.113791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.112979    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"]
Oct 30 21:21:55.122933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.122901    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"]
Oct 30 21:21:55.123253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.123235    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.155201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.155165    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-df078c-openshift-default-dockercfg-dqvc5\""
Oct 30 21:21:55.184060 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:55.184022    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:21:55.208218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.208159    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 30 21:21:55.208395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.208230    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 30 21:21:55.253398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.253359    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-token\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.253610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.253486    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-envoy\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.253610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.253570    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52aca4bf-6016-4c4b-8a5d-29aae1277413-istiod-ca-cert\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.253610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.253594    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-data\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.253823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.253742    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-socket\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.253872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.253822    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-podinfo\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.253961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.253922    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-certs\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.254203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.254185    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59r4k\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-kube-api-access-59r4k\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.254295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.254245    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-credential-socket\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.355518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.355478    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59r4k\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-kube-api-access-59r4k\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.355734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.355555    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-credential-socket\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.355734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.355586    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-token\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.355734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.355663    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-envoy\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.355985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.355859    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52aca4bf-6016-4c4b-8a5d-29aae1277413-istiod-ca-cert\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.355985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.355900    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-data\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.355985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.355950    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-socket\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.356136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.356056    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-podinfo\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.356136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.356097    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-certs\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.356136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.356118    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-credential-socket\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.356603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.356574    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-certs\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.358465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.358443    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-socket\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.358616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.358153    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52aca4bf-6016-4c4b-8a5d-29aae1277413-istiod-ca-cert\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.358974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.358947    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-data\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.361076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.361053    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-envoy\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.361259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.361236    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-podinfo\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.378610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.378536    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-token\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.378772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.378630    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59r4k\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-kube-api-access-59r4k\") pod \"e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") " pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.489321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.488972    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:55.835930 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:55.831466    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:21:55.937701 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:55.936948    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:21:55.964697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:55.955073    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:21:56.402553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:56.402338    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:56.403130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:56.403074    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:56.404639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:56.404612    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:21:56.519938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:56.519413    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"]
Oct 30 21:21:56.531285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:56.531236    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:56.531425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:56.531325    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:56.531425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:56.531370    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:21:57.399749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:57.398865    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" event={"ID":"52aca4bf-6016-4c4b-8a5d-29aae1277413","Type":"ContainerStarted","Data":"ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982"}
Oct 30 21:21:57.399749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:57.398927    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" event={"ID":"52aca4bf-6016-4c4b-8a5d-29aae1277413","Type":"ContainerStarted","Data":"583182509c72498546aeeacf3b42d13a246d14ca17d4996d9d782377322b19f1"}
Oct 30 21:21:57.437277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:57.424959    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podStartSLOduration=3.424940115 podStartE2EDuration="3.424940115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:57.424205049 +0000 UTC m=+1225.773932446" watchObservedRunningTime="2026-10-30 21:21:57.424940115 +0000 UTC m=+1225.774667514"
Oct 30 21:21:57.492030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:57.491997    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:57.493441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:57.493402    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:21:57.493574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:57.493481    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 21:21:58.495558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:58.495521    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:58.956784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:58.956736    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf" podUID="3b1410ef-3273-4786-8af4-2db7aa84a312" containerName="payload-pre-processing" containerID="cri-o://ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807" gracePeriod=25
Oct 30 21:21:59.407069 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.407044    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:59.476811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.476713    4633 generic.go:356] "Generic (PLEG): container finished" podID="3b1410ef-3273-4786-8af4-2db7aa84a312" containerID="ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807" exitCode=0
Oct 30 21:21:59.476811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.476753    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"
Oct 30 21:21:59.476811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.476799    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf" event={"ID":"3b1410ef-3273-4786-8af4-2db7aa84a312","Type":"ContainerDied","Data":"ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807"}
Oct 30 21:21:59.477114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.476858    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf" event={"ID":"3b1410ef-3273-4786-8af4-2db7aa84a312","Type":"ContainerDied","Data":"bafb613651979985e4dc69ed412cb6bbe1175e7a265b27d41022b1e68bd4397b"}
Oct 30 21:21:59.477114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.476885    4633 scope.go:122] "RemoveContainer" containerID="ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807"
Oct 30 21:21:59.477343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.477292    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:59.478408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.478391    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:21:59.487076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.486938    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1410ef-3273-4786-8af4-2db7aa84a312-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1410ef-3273-4786-8af4-2db7aa84a312-plugins-config-volume\") pod \"3b1410ef-3273-4786-8af4-2db7aa84a312\" (UID: \"3b1410ef-3273-4786-8af4-2db7aa84a312\") "
Oct 30 21:21:59.487076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.486988    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1410ef-3273-4786-8af4-2db7aa84a312-kube-api-access-n4dz2\" (UniqueName: \"kubernetes.io/projected/3b1410ef-3273-4786-8af4-2db7aa84a312-kube-api-access-n4dz2\") pod \"3b1410ef-3273-4786-8af4-2db7aa84a312\" (UID: \"3b1410ef-3273-4786-8af4-2db7aa84a312\") "
Oct 30 21:21:59.489377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.487973    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1410ef-3273-4786-8af4-2db7aa84a312-plugins-config-volume" pod "3b1410ef-3273-4786-8af4-2db7aa84a312" (UID: "3b1410ef-3273-4786-8af4-2db7aa84a312"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:59.489911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.489887    4633 scope.go:122] "RemoveContainer" containerID="ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807"
Oct 30 21:21:59.490193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.490161    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1410ef-3273-4786-8af4-2db7aa84a312-kube-api-access-n4dz2" pod "3b1410ef-3273-4786-8af4-2db7aa84a312" (UID: "3b1410ef-3273-4786-8af4-2db7aa84a312"). InnerVolumeSpecName "kube-api-access-n4dz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:59.490280 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:21:59.490262    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807\": container with ID starting with ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807 not found: ID does not exist" containerID="ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807"
Oct 30 21:21:59.490341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.490287    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807"} err="failed to get container status \"ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807\": rpc error: code = NotFound desc = could not find container \"ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807\": container with ID starting with ff2db0e6182bdf8b1dc461e62ab1c212db69c9e683ddd4bf9dee9d8887e21807 not found: ID does not exist"
Oct 30 21:21:59.589489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.589452    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b1410ef-3273-4786-8af4-2db7aa84a312-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:59.589489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.589481    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dz2\" (UniqueName: \"kubernetes.io/projected/3b1410ef-3273-4786-8af4-2db7aa84a312-kube-api-access-n4dz2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:59.880272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.880190    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"]
Oct 30 21:21:59.886172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:21:59.886141    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-7969bf5467-zpnrf"]
Oct 30 21:22:01.712907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.712742    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"]
Oct 30 21:22:01.719798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.713956    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3b1410ef-3273-4786-8af4-2db7aa84a312" containerName="payload-pre-processing"
Oct 30 21:22:01.719798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.713981    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1410ef-3273-4786-8af4-2db7aa84a312" containerName="payload-pre-processing"
Oct 30 21:22:01.719798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.714231    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1410ef-3273-4786-8af4-2db7aa84a312" containerName="payload-pre-processing"
Oct 30 21:22:01.739301 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.738367    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:01.784511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.784473    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09eb50-dockercfg-j4c9d\""
Oct 30 21:22:01.785072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.785047    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09eb50\""
Oct 30 21:22:01.971756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.947504    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgzmk\" (UniqueName: \"kubernetes.io/projected/450f246d-6b20-409e-8438-7d1ce7e777a6-kube-api-access-cgzmk\") pod \"payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc\" (UID: \"450f246d-6b20-409e-8438-7d1ce7e777a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:01.971756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:01.947782    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f246d-6b20-409e-8438-7d1ce7e777a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc\" (UID: \"450f246d-6b20-409e-8438-7d1ce7e777a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:02.049122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:02.049079    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f246d-6b20-409e-8438-7d1ce7e777a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc\" (UID: \"450f246d-6b20-409e-8438-7d1ce7e777a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:02.049590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:02.049567    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgzmk\" (UniqueName: \"kubernetes.io/projected/450f246d-6b20-409e-8438-7d1ce7e777a6-kube-api-access-cgzmk\") pod \"payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc\" (UID: \"450f246d-6b20-409e-8438-7d1ce7e777a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:02.050950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:02.050927    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f246d-6b20-409e-8438-7d1ce7e777a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc\" (UID: \"450f246d-6b20-409e-8438-7d1ce7e777a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:02.085886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:02.085766    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgzmk\" (UniqueName: \"kubernetes.io/projected/450f246d-6b20-409e-8438-7d1ce7e777a6-kube-api-access-cgzmk\") pod \"payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc\" (UID: \"450f246d-6b20-409e-8438-7d1ce7e777a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:02.109904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:02.108550    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1410ef-3273-4786-8af4-2db7aa84a312" path="/var/lib/kubelet/pods/3b1410ef-3273-4786-8af4-2db7aa84a312/volumes"
Oct 30 21:22:02.109904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:02.109066    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"]
Oct 30 21:22:02.320239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:02.318412    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:02.495871 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:02.495838    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:02.864562 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:22:02.864526    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450f246d_6b20_409e_8438_7d1ce7e777a6.slice/crio-0d2200028b828311ff019723363b917649e8946b6376af1d0ab6bbdaf80cc6de WatchSource:0}: Error finding container 0d2200028b828311ff019723363b917649e8946b6376af1d0ab6bbdaf80cc6de: Status 404 returned error can't find the container with id 0d2200028b828311ff019723363b917649e8946b6376af1d0ab6bbdaf80cc6de
Oct 30 21:22:02.871991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:02.871537    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"]
Oct 30 21:22:03.882276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:03.882237    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc" event={"ID":"450f246d-6b20-409e-8438-7d1ce7e777a6","Type":"ContainerStarted","Data":"5eaff34d63ebea24df5bbd07edfe954fe31cfea2209c656323ab698f88884bff"}
Oct 30 21:22:03.885409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:03.885134    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc" event={"ID":"450f246d-6b20-409e-8438-7d1ce7e777a6","Type":"ContainerStarted","Data":"0d2200028b828311ff019723363b917649e8946b6376af1d0ab6bbdaf80cc6de"}
Oct 30 21:22:03.885409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:03.885176    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:03.907957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:03.907902    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc" podStartSLOduration=2.907884992 podStartE2EDuration="2.907884992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:03.905802139 +0000 UTC m=+1232.255529536" watchObservedRunningTime="2026-10-30 21:22:03.907884992 +0000 UTC m=+1232.257612389"
Oct 30 21:22:03.997646 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:03.997595    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:10.406901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.406863    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"]
Oct 30 21:22:10.445778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.439525    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"]
Oct 30 21:22:10.445778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.439653    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.458442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.453785    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-df078c-serving-cert\""
Oct 30 21:22:10.461995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.461960    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.462142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.462024    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.462142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.462066    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbhr\" (UniqueName: \"kubernetes.io/projected/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-kube-api-access-gwbhr\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.462328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.462172    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-metrics-certs\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.611187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.575529    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-metrics-certs\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.611187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.579287    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-metrics-certs\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.611187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.579827    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.611187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.579879    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.611187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.579927    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbhr\" (UniqueName: \"kubernetes.io/projected/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-kube-api-access-gwbhr\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.611187 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:10.580549    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-serving-cert: secret "maas-api-e2e-praxis-a-df078c-serving-cert" not found
Oct 30 21:22:10.611187 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:10.580613    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls podName:bcb4e6ed-33c7-4841-bbf6-9c34211431bf nodeName:}" failed. No retries permitted until 2026-10-30 21:22:11.080593555 +0000 UTC m=+1239.430320929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls") pod "maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28" (UID: "bcb4e6ed-33c7-4841-bbf6-9c34211431bf") : secret "maas-api-e2e-praxis-a-df078c-serving-cert" not found
Oct 30 21:22:10.611187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.581556    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:10.773607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:10.624819    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbhr\" (UniqueName: \"kubernetes.io/projected/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-kube-api-access-gwbhr\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:11.066589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.066552    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": read tcp 10.133.0.2:37646->10.133.0.141:15021: read: connection reset by peer" start-of-body=
Oct 30 21:22:11.066798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.066616    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": read tcp 10.133.0.2:37646->10.133.0.141:15021: read: connection reset by peer"
Oct 30 21:22:11.098796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.095864    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:11.127935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.126928    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:11.383005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.382919    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:11.703659 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:22:11.703617    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb4e6ed_33c7_4841_bbf6_9c34211431bf.slice/crio-1e4e52c36f05857b39d637f9dd71f2fdf22e2f54cddc0b751aa2de1609aff3d4 WatchSource:0}: Error finding container 1e4e52c36f05857b39d637f9dd71f2fdf22e2f54cddc0b751aa2de1609aff3d4: Status 404 returned error can't find the container with id 1e4e52c36f05857b39d637f9dd71f2fdf22e2f54cddc0b751aa2de1609aff3d4
Oct 30 21:22:11.705075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.704985    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"]
Oct 30 21:22:11.963790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.963657    4633 generic.go:356] "Generic (PLEG): container finished" podID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerID="47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11" exitCode=0
Oct 30 21:22:11.963790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.963741    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" event={"ID":"46ce198d-a5bb-4115-bffd-206c9faf4629","Type":"ContainerDied","Data":"47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11"}
Oct 30 21:22:11.964321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.964300    4633 scope.go:122] "RemoveContainer" containerID="47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11"
Oct 30 21:22:11.965536 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.965513    4633 generic.go:356] "Generic (PLEG): container finished" podID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerID="54a06d4426560ba0962609f24997fb522a0becf6e03c89a1a341e35be1d90084" exitCode=0
Oct 30 21:22:11.965638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.965603    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" event={"ID":"730d60c8-aa12-4e39-9fb7-c43aeb8aa656","Type":"ContainerDied","Data":"54a06d4426560ba0962609f24997fb522a0becf6e03c89a1a341e35be1d90084"}
Oct 30 21:22:11.966042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.966025    4633 scope.go:122] "RemoveContainer" containerID="54a06d4426560ba0962609f24997fb522a0becf6e03c89a1a341e35be1d90084"
Oct 30 21:22:11.966210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.966158    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:11.966283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.966259    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:11.966321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.966297    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:11.967146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.967122    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28" event={"ID":"bcb4e6ed-33c7-4841-bbf6-9c34211431bf","Type":"ContainerStarted","Data":"1e4e52c36f05857b39d637f9dd71f2fdf22e2f54cddc0b751aa2de1609aff3d4"}
Oct 30 21:22:11.967874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.967847    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:11.967973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.967915    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:11.968030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:11.967972    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:12.977051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:12.977018    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" event={"ID":"46ce198d-a5bb-4115-bffd-206c9faf4629","Type":"ContainerStarted","Data":"ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675"}
Oct 30 21:22:12.979210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:12.979177    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" event={"ID":"730d60c8-aa12-4e39-9fb7-c43aeb8aa656","Type":"ContainerStarted","Data":"3f2d8a678144a63954cb386bee70cd64caa7ad9c7a053232efb40594f2fd865e"}
Oct 30 21:22:12.980973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:12.980949    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28" event={"ID":"bcb4e6ed-33c7-4841-bbf6-9c34211431bf","Type":"ContainerStarted","Data":"37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee"}
Oct 30 21:22:12.981093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:12.981010    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:13.026722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:13.026640    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28" podStartSLOduration=2.603425734 podStartE2EDuration="3.026622144s" totalImagesPullingTime="423.19641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:13.022024583 +0000 UTC m=+1241.371751980" watchObservedRunningTime="2026-10-30 21:22:13.026622144 +0000 UTC m=+1241.376349542"
Oct 30 21:22:13.415586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:13.415552    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:22:13.519703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:13.513423    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:22:13.519703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:13.513565    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:22:13.519703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:13.513626    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 21:22:13.601741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:13.599850    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:22:13.608694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:13.605797    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 21:22:13.608694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:13.605854    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 21:22:14.176789 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:14.176754    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:14.510496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:14.510406    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:22:14.510496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:14.510484    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 21:22:14.596623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:14.596337    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 21:22:14.596623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:14.596405    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 21:22:14.898464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:14.898388    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:22:15.510237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:15.510198    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:22:15.510650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:15.510266    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 21:22:15.596990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:15.596954    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 21:22:15.597168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:15.597019    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 21:22:15.608202 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:15.608176    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"]
Oct 30 21:22:15.608374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:15.608354    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy" containerID="cri-o://d7a721ffbd2bde2c87e29fdc3e73a9d909690100a3f6ef94ebc56e4de7dedba1" gracePeriod=30
Oct 30 21:22:15.616695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:15.613372    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"]
Oct 30 21:22:15.616695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:15.613624    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy" containerID="cri-o://a3ba535161755c60a2ba89b7a5f2660f87dd90e1a30fdd3774561180a0a1d038" gracePeriod=30
Oct 30 21:22:16.510245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:16.510200    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:22:16.510766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:16.510282    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 21:22:16.634911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:16.631812    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 21:22:16.634911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:16.631876    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 21:22:16.836751 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:16.803021    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:17.533542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:17.532987    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:22:17.533542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:17.533056    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 21:22:17.613698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:17.613233    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 21:22:17.613698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:17.613312    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 21:22:18.511984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:18.511941    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:22:18.512290 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:18.512259    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 21:22:18.597197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:18.597158    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 21:22:18.597836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:18.597785    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 21:22:19.005414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:19.003750    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:22:19.511082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:19.511041    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:22:19.511271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:19.511132    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 21:22:19.599896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:19.599860    4633 patch_prober.go:29] interesting pod/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 30 21:22:19.600467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:19.600433    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 30 21:22:20.465693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.462695    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"]
Oct 30 21:22:20.478541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.476724    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"]
Oct 30 21:22:20.478541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.476861    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.481796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.481769    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-50d070-openshift-default-dockercfg-q8dkn\""
Oct 30 21:22:20.526705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.521830    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:22:20.526999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.526962    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 21:22:20.579716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.567682    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-envoy\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.606694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.606640    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ss7m\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-kube-api-access-4ss7m\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.607214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.607192    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-credential-socket\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.607478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.607440    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-socket\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.607844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.607770    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29883ff-2844-4d5f-b016-9e87407390fc-istiod-ca-cert\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.608116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.608013    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29883ff-2844-4d5f-b016-9e87407390fc-istio-podinfo\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.623112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.623002    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-data\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.623396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.623082    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-certs\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.623577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.623519    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-istio-token\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.738021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.726009    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-data\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.738021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.726058    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-certs\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.738021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.726110    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-istio-token\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.738021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.726278    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-envoy\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.738021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.726312    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ss7m\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-kube-api-access-4ss7m\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.738021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.726342    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-credential-socket\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.738021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.726396    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-socket\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.738021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.729123    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-credential-socket\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.764245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.726477    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29883ff-2844-4d5f-b016-9e87407390fc-istiod-ca-cert\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.796342 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.796310    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29883ff-2844-4d5f-b016-9e87407390fc-istio-podinfo\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.877385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.802486    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-socket\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.877385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.823862    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-certs\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.877385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.834379    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-data\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.877385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.835193    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29883ff-2844-4d5f-b016-9e87407390fc-istiod-ca-cert\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.890242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.890205    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-envoy\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.952502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.952467    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29883ff-2844-4d5f-b016-9e87407390fc-istio-podinfo\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:20.956917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:20.953228    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-istio-token\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:21.067499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.067394    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ss7m\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-kube-api-access-4ss7m\") pod \"e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") " pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:21.136272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.135979    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:21.362825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.362004    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:22:21.403967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.397496    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:22:21.443138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.439887    4633 generic.go:356] "Generic (PLEG): container finished" podID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerID="a3ba535161755c60a2ba89b7a5f2660f87dd90e1a30fdd3774561180a0a1d038" exitCode=0
Oct 30 21:22:21.443138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.440014    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" event={"ID":"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69","Type":"ContainerDied","Data":"a3ba535161755c60a2ba89b7a5f2660f87dd90e1a30fdd3774561180a0a1d038"}
Oct 30 21:22:21.476422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.475308    4633 generic.go:356] "Generic (PLEG): container finished" podID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerID="d7a721ffbd2bde2c87e29fdc3e73a9d909690100a3f6ef94ebc56e4de7dedba1" exitCode=0
Oct 30 21:22:21.476422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.476348    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" event={"ID":"ae6bab83-0de4-4758-bd89-53f27be63c2d","Type":"ContainerDied","Data":"d7a721ffbd2bde2c87e29fdc3e73a9d909690100a3f6ef94ebc56e4de7dedba1"}
Oct 30 21:22:21.526699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.519975    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:22:21.526699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.521592    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:22:21.526699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.525922    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:22:21.629283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.626978    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"]
Oct 30 21:22:21.657157 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:22:21.657040    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29883ff_2844_4d5f_b016_9e87407390fc.slice/crio-2d3b12f3bda39667aa7d9b46f8b0c43a3f8d455c97e345bd280143f33ac52e9f WatchSource:0}: Error finding container 2d3b12f3bda39667aa7d9b46f8b0c43a3f8d455c97e345bd280143f33ac52e9f: Status 404 returned error can't find the container with id 2d3b12f3bda39667aa7d9b46f8b0c43a3f8d455c97e345bd280143f33ac52e9f
Oct 30 21:22:21.660581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.660532    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:21.660820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.660644    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:21.660820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.660772    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:21.852325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.852298    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:22:21.870882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.870854    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:22:21.974280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.974249    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-token\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-token\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.974970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.974945    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-certs\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.975157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975140    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-data\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.975347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975317    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-certs" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.975691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975580    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-data" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.975691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975333    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-kube-api-access-w6kmv\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-kube-api-access-w6kmv\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.975888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975748    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istiod-ca-cert\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.975888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975794    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-certs\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.975888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975826    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-socket\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.975888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975881    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-credential-socket\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.976109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.975996    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6bab83-0de4-4758-bd89-53f27be63c2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bab83-0de4-4758-bd89-53f27be63c2d-istiod-ca-cert\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.976109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976046    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-data\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.976109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976084    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-envoy\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.976285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976176    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-token\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-token\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.976285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976213    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-kube-api-access-g68sh\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-kube-api-access-g68sh\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.976285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976241    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-podinfo\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.976499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976300    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-socket\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.976499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976346    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-envoy\") pod \"ae6bab83-0de4-4758-bd89-53f27be63c2d\" (UID: \"ae6bab83-0de4-4758-bd89-53f27be63c2d\") "
Oct 30 21:22:21.976499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976388    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-credential-socket\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.976651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.976504    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-podinfo\") pod \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\" (UID: \"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69\") "
Oct 30 21:22:21.977045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.977024    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-token" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:21.977406 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.977388    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:21.977513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.977501    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:21.977600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.977589    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:21.977722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.977629    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-socket" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.977991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.977973    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-certs" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.978075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.978019    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istiod-ca-cert" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:21.978487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.978469    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6bab83-0de4-4758-bd89-53f27be63c2d-istiod-ca-cert" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:21.978848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.978830    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-credential-socket" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.979121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.979104    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-data" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.980538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.980511    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-token" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:21.980827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.980803    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-socket" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.981053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.980924    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-kube-api-access-w6kmv" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "kube-api-access-w6kmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:21.981264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.981240    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-credential-socket" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.981339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.981322    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-podinfo" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:21.984121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.984091    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-podinfo" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:21.984121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.984113    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-kube-api-access-g68sh" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "kube-api-access-g68sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:21.984356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.984332    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-envoy" pod "ae6bab83-0de4-4758-bd89-53f27be63c2d" (UID: "ae6bab83-0de4-4758-bd89-53f27be63c2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:21.984783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:21.984760    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-envoy" pod "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" (UID: "3a0d2768-381d-4fa1-9d8f-d658fd1d3f69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.101948    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.101988    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g68sh\" (UniqueName: \"kubernetes.io/projected/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-kube-api-access-g68sh\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102005    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102019    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102037    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102051    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102071    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102085    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6kmv\" (UniqueName: \"kubernetes.io/projected/ae6bab83-0de4-4758-bd89-53f27be63c2d-kube-api-access-w6kmv\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102098    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102109    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102121    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102138    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102149    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bab83-0de4-4758-bd89-53f27be63c2d-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102161    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6bab83-0de4-4758-bd89-53f27be63c2d-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.103694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.102175    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:22.486379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.486330    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" event={"ID":"c29883ff-2844-4d5f-b016-9e87407390fc","Type":"ContainerStarted","Data":"441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650"}
Oct 30 21:22:22.486593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.486439    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" event={"ID":"c29883ff-2844-4d5f-b016-9e87407390fc","Type":"ContainerStarted","Data":"2d3b12f3bda39667aa7d9b46f8b0c43a3f8d455c97e345bd280143f33ac52e9f"}
Oct 30 21:22:22.488266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.488241    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv" event={"ID":"3a0d2768-381d-4fa1-9d8f-d658fd1d3f69","Type":"ContainerDied","Data":"86bbed78a41d43e9092b722dd2c56cc84c0d6f10e14f30ffca59b0d244680d4b"}
Oct 30 21:22:22.488266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.488256    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"
Oct 30 21:22:22.488450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.488285    4633 scope.go:122] "RemoveContainer" containerID="a3ba535161755c60a2ba89b7a5f2660f87dd90e1a30fdd3774561180a0a1d038"
Oct 30 21:22:22.490060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.489969    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"
Oct 30 21:22:22.490060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.489995    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st" event={"ID":"ae6bab83-0de4-4758-bd89-53f27be63c2d","Type":"ContainerDied","Data":"962a8b1cfa38f42c8143354167081de306e9628b87ff13ab10072f34337c492e"}
Oct 30 21:22:22.503399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.503373    4633 scope.go:122] "RemoveContainer" containerID="d7a721ffbd2bde2c87e29fdc3e73a9d909690100a3f6ef94ebc56e4de7dedba1"
Oct 30 21:22:22.522991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.522923    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podStartSLOduration=2.522902205 podStartE2EDuration="2.522902205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:22.507930431 +0000 UTC m=+1250.857657852" watchObservedRunningTime="2026-10-30 21:22:22.522902205 +0000 UTC m=+1250.872629601"
Oct 30 21:22:22.554098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.554072    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"]
Oct 30 21:22:22.558749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.558724    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-pq8st"]
Oct 30 21:22:22.563720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.563691    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"]
Oct 30 21:22:22.570359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:22.570337    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-79xnv"]
Oct 30 21:22:23.137916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:23.137879    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:23.145119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:23.145093    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:23.607417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:23.599774    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:23.614285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:23.613951    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:22:23.847759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:23.838585    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" path="/var/lib/kubelet/pods/3a0d2768-381d-4fa1-9d8f-d658fd1d3f69/volumes"
Oct 30 21:22:23.847759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:23.839470    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" path="/var/lib/kubelet/pods/ae6bab83-0de4-4758-bd89-53f27be63c2d/volumes"
Oct 30 21:22:24.193215 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:24.193188    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:24.800771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:24.800737    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-nb8q8_d3af9a3e-eac9-4673-b9da-fc0dc10bab32/payload-processing/0.log"
Oct 30 21:22:25.043270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:25.043241    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-nb8q8_d3af9a3e-eac9-4673-b9da-fc0dc10bab32/payload-processing/0.log"
Oct 30 21:22:25.044126 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:25.044083    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:22:26.293799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.293764    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"]
Oct 30 21:22:26.295564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.295534    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy"
Oct 30 21:22:26.295564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.295563    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy"
Oct 30 21:22:26.295864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.295601    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy"
Oct 30 21:22:26.295864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.295609    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy"
Oct 30 21:22:26.295864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.295818    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6bab83-0de4-4758-bd89-53f27be63c2d" containerName="istio-proxy"
Oct 30 21:22:26.295864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.295837    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0d2768-381d-4fa1-9d8f-d658fd1d3f69" containerName="istio-proxy"
Oct 30 21:22:26.302868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.302823    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:26.329210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.329176    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-50d070\""
Oct 30 21:22:26.351270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.351239    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"]
Oct 30 21:22:26.401322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.401282    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"]
Oct 30 21:22:26.408454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.408424    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh5k6\" (UniqueName: \"kubernetes.io/projected/e2699055-92f7-4648-b876-08fcf4de3f25-kube-api-access-vh5k6\") pod \"payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw\" (UID: \"e2699055-92f7-4648-b876-08fcf4de3f25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:26.408847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.408823    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2699055-92f7-4648-b876-08fcf4de3f25-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw\" (UID: \"e2699055-92f7-4648-b876-08fcf4de3f25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:26.413755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.409827    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:26.458717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.432146    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-50d070-dockercfg-bpnfp\""
Oct 30 21:22:26.463691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.461846    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"]
Oct 30 21:22:26.513612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.513568    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2699055-92f7-4648-b876-08fcf4de3f25-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw\" (UID: \"e2699055-92f7-4648-b876-08fcf4de3f25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:26.546477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.546389    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5k6\" (UniqueName: \"kubernetes.io/projected/e2699055-92f7-4648-b876-08fcf4de3f25-kube-api-access-vh5k6\") pod \"payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw\" (UID: \"e2699055-92f7-4648-b876-08fcf4de3f25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:26.568150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.568111    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjvg\" (UniqueName: \"kubernetes.io/projected/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-kube-api-access-pnjvg\") pod \"payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76\" (UID: \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:26.568634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.568609    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76\" (UID: \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:26.569059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.522621    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2699055-92f7-4648-b876-08fcf4de3f25-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw\" (UID: \"e2699055-92f7-4648-b876-08fcf4de3f25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:26.658695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.658025    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh5k6\" (UniqueName: \"kubernetes.io/projected/e2699055-92f7-4648-b876-08fcf4de3f25-kube-api-access-vh5k6\") pod \"payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw\" (UID: \"e2699055-92f7-4648-b876-08fcf4de3f25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:26.673927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.672953    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:26.675276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.675242    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76\" (UID: \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:26.756706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.742152    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjvg\" (UniqueName: \"kubernetes.io/projected/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-kube-api-access-pnjvg\") pod \"payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76\" (UID: \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:26.756706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.721003    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76\" (UID: \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:26.785073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.785016    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjvg\" (UniqueName: \"kubernetes.io/projected/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-kube-api-access-pnjvg\") pod \"payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76\" (UID: \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:26.881093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:26.880815    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:27.342311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:27.341927    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"]
Oct 30 21:22:27.568406 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:27.568376    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"]
Oct 30 21:22:27.568544 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:22:27.568503    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf82d1dd_75e3_4bc8_847d_40fd2ef35f73.slice/crio-e59261dd28e227b3704ee874cff08c7e0de750ab48beb5649b4990d792ead2ee WatchSource:0}: Error finding container e59261dd28e227b3704ee874cff08c7e0de750ab48beb5649b4990d792ead2ee: Status 404 returned error can't find the container with id e59261dd28e227b3704ee874cff08c7e0de750ab48beb5649b4990d792ead2ee
Oct 30 21:22:27.668385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:27.668349    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76" event={"ID":"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73","Type":"ContainerStarted","Data":"e59261dd28e227b3704ee874cff08c7e0de750ab48beb5649b4990d792ead2ee"}
Oct 30 21:22:27.670327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:27.670301    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw" event={"ID":"e2699055-92f7-4648-b876-08fcf4de3f25","Type":"ContainerStarted","Data":"01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c"}
Oct 30 21:22:27.670453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:27.670351    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw" event={"ID":"e2699055-92f7-4648-b876-08fcf4de3f25","Type":"ContainerStarted","Data":"a1a26de49b0b9c9079651778cb694a3e9193478ec89582577ea17bfa12da8ad8"}
Oct 30 21:22:27.670453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:27.670399    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:27.692439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:27.692381    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw" podStartSLOduration=1.6923623349999999 podStartE2EDuration="1.692362335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:27.690016412 +0000 UTC m=+1256.039743810" watchObservedRunningTime="2026-10-30 21:22:27.692362335 +0000 UTC m=+1256.042089731"
Oct 30 21:22:28.603466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:28.603427    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"]
Oct 30 21:22:28.603976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:28.603707    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy" containerID="cri-o://438a8562538b4b0ddda04d58f6263b61c914ad700647c0d65e0c897da7ef15e4" gracePeriod=30
Oct 30 21:22:28.681398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:28.681363    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76" event={"ID":"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73","Type":"ContainerStarted","Data":"359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c"}
Oct 30 21:22:28.681605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:28.681443    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:28.702634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:28.702580    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76" podStartSLOduration=2.702565861 podStartE2EDuration="2.702565861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:28.699697118 +0000 UTC m=+1257.049424523" watchObservedRunningTime="2026-10-30 21:22:28.702565861 +0000 UTC m=+1257.052293258"
Oct 30 21:22:31.716962 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:31.716883    4633 generic.go:356] "Generic (PLEG): container finished" podID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerID="62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78" exitCode=0
Oct 30 21:22:31.717377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:31.716962    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" event={"ID":"020842ff-b53f-47ce-a40f-72e4e56f2c2c","Type":"ContainerDied","Data":"62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78"}
Oct 30 21:22:31.717454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:31.717439    4633 scope.go:122] "RemoveContainer" containerID="62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78"
Oct 30 21:22:31.719347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:31.719314    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:31.719453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:31.719395    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:31.719453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:31.719445    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:31.891193 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:31.891162    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@12-24831-0.service\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:32.738700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:32.738641    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" event={"ID":"020842ff-b53f-47ce-a40f-72e4e56f2c2c","Type":"ContainerStarted","Data":"34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901"}
Oct 30 21:22:32.945436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:32.945391    4633 scope.go:122] "RemoveContainer" containerID="783e9f2384cec430445be0bf8800dbad26c29380d914b8ac604a183f0823d700"
Oct 30 21:22:32.963092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:32.963068    4633 scope.go:122] "RemoveContainer" containerID="4c4c03f0bf672f146b15b660f5f68248910afdb9237fe1765edfca83ab1529ac"
Oct 30 21:22:33.270394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:33.270306    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:22:33.272336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:33.272310    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:22:33.272463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:33.272359    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:22:33.748506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:33.748471    4633 generic.go:356] "Generic (PLEG): container finished" podID="31437c22-0870-430e-8ab4-195e4b44d622" containerID="438a8562538b4b0ddda04d58f6263b61c914ad700647c0d65e0c897da7ef15e4" exitCode=0
Oct 30 21:22:33.748840 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:33.748537    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" event={"ID":"31437c22-0870-430e-8ab4-195e4b44d622","Type":"ContainerDied","Data":"438a8562538b4b0ddda04d58f6263b61c914ad700647c0d65e0c897da7ef15e4"}
Oct 30 21:22:33.962337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:33.962312    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:22:34.069335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069293    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-envoy\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.069541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069485    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-data\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.069620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069556    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-kube-api-access-td9sr\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-kube-api-access-td9sr\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.069620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069588    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-credential-socket\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.069760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069642    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31437c22-0870-430e-8ab4-195e4b44d622-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31437c22-0870-430e-8ab4-195e4b44d622-istio-podinfo\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.069760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069689    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-istio-token\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-istio-token\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.069760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069724    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-socket\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.069927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069881    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-certs\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.069927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.069917    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31437c22-0870-430e-8ab4-195e4b44d622-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31437c22-0870-430e-8ab4-195e4b44d622-istiod-ca-cert\") pod \"31437c22-0870-430e-8ab4-195e4b44d622\" (UID: \"31437c22-0870-430e-8ab4-195e4b44d622\") "
Oct 30 21:22:34.071151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.071123    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31437c22-0870-430e-8ab4-195e4b44d622-istiod-ca-cert" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:34.072071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.072048    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-socket" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:34.080663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.072985    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-data" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:34.080663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.073545    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-certs" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:34.080663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.073608    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-credential-socket" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:34.080663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.073904    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-envoy" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:34.080663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.076229    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-kube-api-access-td9sr" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "kube-api-access-td9sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:34.080663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.078030    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31437c22-0870-430e-8ab4-195e4b44d622-istio-podinfo" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:34.080663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.080213    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-istio-token" pod "31437c22-0870-430e-8ab4-195e4b44d622" (UID: "31437c22-0870-430e-8ab4-195e4b44d622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177104    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177147    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31437c22-0870-430e-8ab4-195e4b44d622-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177162    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177176    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177191    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9sr\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-kube-api-access-td9sr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177207    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177221    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31437c22-0870-430e-8ab4-195e4b44d622-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177236    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31437c22-0870-430e-8ab4-195e4b44d622-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.178760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.177249    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31437c22-0870-430e-8ab4-195e4b44d622-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:34.271388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.271355    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:22:34.271564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.271412    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:22:34.708790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.708753    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:22:34.839696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.832294    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs" event={"ID":"31437c22-0870-430e-8ab4-195e4b44d622","Type":"ContainerDied","Data":"625ecaf7a7579514b289f9e1e5785342d4e0469efa68da5373317d2c72c753fa"}
Oct 30 21:22:34.839696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.832364    4633 scope.go:122] "RemoveContainer" containerID="438a8562538b4b0ddda04d58f6263b61c914ad700647c0d65e0c897da7ef15e4"
Oct 30 21:22:34.839696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.832529    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"
Oct 30 21:22:34.955968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.955934    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"]
Oct 30 21:22:34.973276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:34.973149    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-4rxvs"]
Oct 30 21:22:35.271596 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:35.271512    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:22:35.271596 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:35.271579    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:22:35.853697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:35.849466    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31437c22-0870-430e-8ab4-195e4b44d622" path="/var/lib/kubelet/pods/31437c22-0870-430e-8ab4-195e4b44d622/volumes"
Oct 30 21:22:36.278702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:36.274173    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:22:36.278702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:36.274243    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:22:37.271172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:37.271121    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:22:37.271601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:37.271184    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:22:38.094449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.094412    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"]
Oct 30 21:22:38.099103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.098528    4633 cpu_manager.go:414] "containerMap: removing container" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy"
Oct 30 21:22:38.099103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.098553    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy"
Oct 30 21:22:38.099103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.098825    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="31437c22-0870-430e-8ab4-195e4b44d622" containerName="istio-proxy"
Oct 30 21:22:38.108965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.108943    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.141204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.141158    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-50d070-serving-cert\""
Oct 30 21:22:38.172059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.169131    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"]
Oct 30 21:22:38.191717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.191662    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-metrics-certs\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.191928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.191908    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.192169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.192150    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2plsq\" (UniqueName: \"kubernetes.io/projected/cb63490a-6a55-4e83-bdea-4418722d07cd-kube-api-access-2plsq\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.192243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.192184    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63490a-6a55-4e83-bdea-4418722d07cd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.213713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.210785    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:22:38.333690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.326691    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2plsq\" (UniqueName: \"kubernetes.io/projected/cb63490a-6a55-4e83-bdea-4418722d07cd-kube-api-access-2plsq\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.333690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.326744    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63490a-6a55-4e83-bdea-4418722d07cd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.333690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.326898    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-metrics-certs\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.333690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.327160    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.333690 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:38.327362    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-serving-cert: secret "maas-api-e2e-praxis-b-50d070-serving-cert" not found
Oct 30 21:22:38.333690 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:38.327450    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls podName:cb63490a-6a55-4e83-bdea-4418722d07cd nodeName:}" failed. No retries permitted until 2026-10-30 21:22:38.827423893 +0000 UTC m=+1267.177151292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls") pod "maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8" (UID: "cb63490a-6a55-4e83-bdea-4418722d07cd") : secret "maas-api-e2e-praxis-b-50d070-serving-cert" not found
Oct 30 21:22:38.333690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.328720    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63490a-6a55-4e83-bdea-4418722d07cd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.373286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.373205    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:22:38.373514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.373490    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:22:38.406709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.388712    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-metrics-certs\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.420700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.410525    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2plsq\" (UniqueName: \"kubernetes.io/projected/cb63490a-6a55-4e83-bdea-4418722d07cd-kube-api-access-2plsq\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:38.475026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.474989    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"]
Oct 30 21:22:38.475535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.475485    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy" containerID="cri-o://1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece" gracePeriod=30
Oct 30 21:22:38.733696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.730954    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:22:38.854696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.854100    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:39.013823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:38.995220    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:39.184702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:39.178404    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:39.302252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:39.302176    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Oct 30 21:22:39.302481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:39.302457    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Oct 30 21:22:39.653014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:39.648789    4633 patch_prober.go:29] interesting pod/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:22:39.653014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:39.648855    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:22:39.733707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:39.713903    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:22:40.050764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:40.049727    4633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:22:40.051326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:40.051301    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"]
Oct 30 21:22:40.256027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:40.255989    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8" event={"ID":"cb63490a-6a55-4e83-bdea-4418722d07cd","Type":"ContainerStarted","Data":"7c4cd67ded228521c8c3ac2a5c8c54ce8e820d70786f9f93f4edf303ae42cc76"}
Oct 30 21:22:40.275500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:40.275468    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:22:40.276479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:40.276455    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:22:41.368837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:41.368794    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8" event={"ID":"cb63490a-6a55-4e83-bdea-4418722d07cd","Type":"ContainerStarted","Data":"2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888"}
Oct 30 21:22:41.393960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:41.393889    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8" podStartSLOduration=2.860611624 podStartE2EDuration="3.393869978s" totalImagesPullingTime="533.258354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:41.390429497 +0000 UTC m=+1269.740156905" watchObservedRunningTime="2026-10-30 21:22:41.393869978 +0000 UTC m=+1269.743597375"
Oct 30 21:22:42.376731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:42.376703    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:43.863684 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.863650    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:22:43.921552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.921504    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-credential-socket\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.921921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.921626    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-socket\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.921921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.921730    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-data\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.921921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.921817    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-certs\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.922153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.921818    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-credential-socket" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:43.922153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.922039    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-data" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:43.922153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.922045    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-socket" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:43.922153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.922087    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-certs" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:43.922501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.922194    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-token\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-token\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.922501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.922274    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-kube-api-access-8h4cd\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-kube-api-access-8h4cd\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.922501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.922331    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-envoy\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.922761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.922741    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01d8427-ce69-4098-b6c9-cf2eb7530091-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01d8427-ce69-4098-b6c9-cf2eb7530091-istiod-ca-cert\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.923187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.923164    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01d8427-ce69-4098-b6c9-cf2eb7530091-istiod-ca-cert" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:43.923292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.923274    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-podinfo\") pod \"f01d8427-ce69-4098-b6c9-cf2eb7530091\" (UID: \"f01d8427-ce69-4098-b6c9-cf2eb7530091\") "
Oct 30 21:22:43.925027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.925000    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-envoy" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:43.926043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.926002    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-token" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:43.926466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.926441    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f01d8427-ce69-4098-b6c9-cf2eb7530091-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:43.926568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.926469    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:43.926568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.926482    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:43.926568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.926495    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:43.926568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.926508    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:43.926568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.926519    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:43.926568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.926532    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:43.927075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.927048    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-kube-api-access-8h4cd" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "kube-api-access-8h4cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:43.927166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:43.927092    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-podinfo" pod "f01d8427-ce69-4098-b6c9-cf2eb7530091" (UID: "f01d8427-ce69-4098-b6c9-cf2eb7530091"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:44.028131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.028098    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h4cd\" (UniqueName: \"kubernetes.io/projected/f01d8427-ce69-4098-b6c9-cf2eb7530091-kube-api-access-8h4cd\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.028131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.028128    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f01d8427-ce69-4098-b6c9-cf2eb7530091-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.395344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.395302    4633 generic.go:356] "Generic (PLEG): container finished" podID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerID="1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece" exitCode=0
Oct 30 21:22:44.395560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.395384    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"
Oct 30 21:22:44.395560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.395383    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" event={"ID":"f01d8427-ce69-4098-b6c9-cf2eb7530091","Type":"ContainerDied","Data":"1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece"}
Oct 30 21:22:44.395560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.395466    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx" event={"ID":"f01d8427-ce69-4098-b6c9-cf2eb7530091","Type":"ContainerDied","Data":"cb46d44ae9a1fc968bcde9f354772dd4569de8edfd7541d5e5957d4516a87d25"}
Oct 30 21:22:44.395560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.395489    4633 scope.go:122] "RemoveContainer" containerID="1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece"
Oct 30 21:22:44.408868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.408841    4633 scope.go:122] "RemoveContainer" containerID="b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b"
Oct 30 21:22:44.421903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.421872    4633 scope.go:122] "RemoveContainer" containerID="1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece"
Oct 30 21:22:44.425584 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:44.424868    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece\": container with ID starting with 1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece not found: ID does not exist" containerID="1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece"
Oct 30 21:22:44.425584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.424910    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece"} err="failed to get container status \"1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece\": rpc error: code = NotFound desc = could not find container \"1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece\": container with ID starting with 1ad7ba79f888ef97c4a08a2eef7cfdcd64f7f2749ebc53343709313ac9ea3ece not found: ID does not exist"
Oct 30 21:22:44.425584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.424936    4633 scope.go:122] "RemoveContainer" containerID="b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b"
Oct 30 21:22:44.425584 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:44.425380    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b\": container with ID starting with b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b not found: ID does not exist" containerID="b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b"
Oct 30 21:22:44.425584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.425413    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b"} err="failed to get container status \"b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b\": rpc error: code = NotFound desc = could not find container \"b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b\": container with ID starting with b632f5eae6d64c1847e8896f7502e287992bf3b6a80bbe8e4adeb694b8c7e19b not found: ID does not exist"
Oct 30 21:22:44.429012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.428990    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"]
Oct 30 21:22:44.436471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:44.436444    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dbc08f-openshift-default-847cc86bc9-l4svx"]
Oct 30 21:22:45.537341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:45.537301    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:22:45.537957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:45.537383    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:22:45.792703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:45.792618    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" path="/var/lib/kubelet/pods/f01d8427-ce69-4098-b6c9-cf2eb7530091/volumes"
Oct 30 21:22:48.392047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.392012    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:22:48.432340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.432266    4633 generic.go:356] "Generic (PLEG): container finished" podID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerID="ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982" exitCode=0
Oct 30 21:22:48.432340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.432334    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" event={"ID":"52aca4bf-6016-4c4b-8a5d-29aae1277413","Type":"ContainerDied","Data":"ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982"}
Oct 30 21:22:48.432946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.432876    4633 scope.go:122] "RemoveContainer" containerID="ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982"
Oct 30 21:22:48.434865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.434831    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:48.435006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.434910    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:48.435006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.434949    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:22:48.664832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.663471    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"]
Oct 30 21:22:48.664832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:48.663741    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy" containerID="cri-o://df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916" gracePeriod=30
Oct 30 21:22:49.474208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:49.474158    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" event={"ID":"52aca4bf-6016-4c4b-8a5d-29aae1277413","Type":"ContainerStarted","Data":"37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95"}
Oct 30 21:22:49.493259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:49.493228    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:22:49.494328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:49.494303    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:22:49.497693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:49.494528    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 21:22:50.489937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:50.489904    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:22:50.490389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:50.489972    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 21:22:51.489647 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:51.489618    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:22:51.489877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:51.489692    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 21:22:52.490056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:52.490020    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:22:52.490484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:52.490080    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 21:22:53.489878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:53.489841    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:22:53.490067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:53.489896    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 21:22:54.010760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.010610    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:22:54.152539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152506    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-socket\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.152728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152574    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41a2699d-77ea-4e27-ac72-8428c096e798-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a2699d-77ea-4e27-ac72-8428c096e798-istio-podinfo\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.152728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152616    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-istio-token\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-istio-token\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.152728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152634    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-kube-api-access-mxtjx\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-kube-api-access-mxtjx\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.152728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152721    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a2699d-77ea-4e27-ac72-8428c096e798-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a2699d-77ea-4e27-ac72-8428c096e798-istiod-ca-cert\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.152872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152771    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-envoy\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.152872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152824    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-credential-socket\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.152872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152847    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-certs\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.152974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.152959    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-data\") pod \"41a2699d-77ea-4e27-ac72-8428c096e798\" (UID: \"41a2699d-77ea-4e27-ac72-8428c096e798\") "
Oct 30 21:22:54.153451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.153407    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-socket" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:54.153875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.153626    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-data" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:54.153875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.153636    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.153875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.153647    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-credential-socket" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:54.153875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.153729    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a2699d-77ea-4e27-ac72-8428c096e798-istiod-ca-cert" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:54.153875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.153838    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-certs" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:54.156016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.155995    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-kube-api-access-mxtjx" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "kube-api-access-mxtjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:54.156247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.156231    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41a2699d-77ea-4e27-ac72-8428c096e798-istio-podinfo" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:54.158310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.158288    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-envoy" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:54.158410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.158371    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-istio-token" pod "41a2699d-77ea-4e27-ac72-8428c096e798" (UID: "41a2699d-77ea-4e27-ac72-8428c096e798"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:54.255066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.255027    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.255066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.255066    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.255289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.255083    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.255289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.255096    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a2699d-77ea-4e27-ac72-8428c096e798-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.255289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.255109    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a2699d-77ea-4e27-ac72-8428c096e798-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.255289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.255123    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.255289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.255138    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxtjx\" (UniqueName: \"kubernetes.io/projected/41a2699d-77ea-4e27-ac72-8428c096e798-kube-api-access-mxtjx\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.255289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.255153    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a2699d-77ea-4e27-ac72-8428c096e798-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.490134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.490091    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:22:54.490354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.490172    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 21:22:54.552629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.552579    4633 generic.go:356] "Generic (PLEG): container finished" podID="41a2699d-77ea-4e27-ac72-8428c096e798" containerID="df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916" exitCode=0
Oct 30 21:22:54.552885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.552840    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" event={"ID":"41a2699d-77ea-4e27-ac72-8428c096e798","Type":"ContainerDied","Data":"df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916"}
Oct 30 21:22:54.552982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.552945    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh" event={"ID":"41a2699d-77ea-4e27-ac72-8428c096e798","Type":"ContainerDied","Data":"2f1cf6cd7b80e70b3981a9821897f1a5dbc05571459994786c5898a9610d88ee"}
Oct 30 21:22:54.553050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.552980    4633 scope.go:122] "RemoveContainer" containerID="df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916"
Oct 30 21:22:54.553185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.553167    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"
Oct 30 21:22:54.569520 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.569492    4633 scope.go:122] "RemoveContainer" containerID="81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd"
Oct 30 21:22:54.581621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.581599    4633 scope.go:122] "RemoveContainer" containerID="df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916"
Oct 30 21:22:54.581982 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:54.581952    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916\": container with ID starting with df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916 not found: ID does not exist" containerID="df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916"
Oct 30 21:22:54.582076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.581998    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916"} err="failed to get container status \"df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916\": rpc error: code = NotFound desc = could not find container \"df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916\": container with ID starting with df11177cab1bae87395275b0bab15f1de16bb8b39a9d1655a95f40e835fcd916 not found: ID does not exist"
Oct 30 21:22:54.582076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.582022    4633 scope.go:122] "RemoveContainer" containerID="81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd"
Oct 30 21:22:54.582358 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:54.582331    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd\": container with ID starting with 81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd not found: ID does not exist" containerID="81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd"
Oct 30 21:22:54.582445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.582367    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd"} err="failed to get container status \"81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd\": rpc error: code = NotFound desc = could not find container \"81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd\": container with ID starting with 81ed404939c42ed8977f41ba8ddd8b8a0e00f3dec8b5897e55ee9b691f985dbd not found: ID does not exist"
Oct 30 21:22:54.593431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.593402    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"]
Oct 30 21:22:54.597580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:54.597558    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2a55fb-openshift-default-76cbcb78fc-dk7lh"]
Oct 30 21:22:55.280744 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:55.274215    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:22:55.489940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:55.489907    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 21:22:55.490113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:55.489971    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 21:22:55.793588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:55.793552    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" path="/var/lib/kubelet/pods/41a2699d-77ea-4e27-ac72-8428c096e798/volumes"
Oct 30 21:22:56.494541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:56.494511    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:22:56.495023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:56.495005    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:22:56.495978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:56.495950    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:22:57.779483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:57.779447    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"]
Oct 30 21:22:57.780325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:57.780274    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh" podUID="e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" containerName="maas-api" containerID="cri-o://bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4" gracePeriod=30
Oct 30 21:22:58.261788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.261381    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:22:58.313339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.313301    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-kube-api-access-dhggs\" (UniqueName: \"kubernetes.io/projected/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-kube-api-access-dhggs\") pod \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") "
Oct 30 21:22:58.320692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.313803    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-openshift-service-ca\") pod \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") "
Oct 30 21:22:58.330770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.320968    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls\") pod \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") "
Oct 30 21:22:58.348750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.338172    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-openshift-service-ca" pod "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" (UID: "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:58.348750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.338501    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-metrics-certs\") pod \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\" (UID: \"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40\") "
Oct 30 21:22:58.348750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.347976    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls" pod "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" (UID: "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:58.348750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.348460    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-metrics-certs" pod "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" (UID: "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:58.353889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.353813    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.380559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.378451    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-kube-api-access-dhggs" pod "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" (UID: "e4001d79-f0a5-4dbe-bf55-d8ec626b2c40"). InnerVolumeSpecName "kube-api-access-dhggs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:58.475777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.475165    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhggs\" (UniqueName: \"kubernetes.io/projected/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-kube-api-access-dhggs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.475777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.475210    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.475777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.475239    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.618213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.612269    4633 generic.go:356] "Generic (PLEG): container finished" podID="e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" containerID="bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4" exitCode=0
Oct 30 21:22:58.618213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.612361    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh" event={"ID":"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40","Type":"ContainerDied","Data":"bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4"}
Oct 30 21:22:58.618213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.612589    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh" event={"ID":"e4001d79-f0a5-4dbe-bf55-d8ec626b2c40","Type":"ContainerDied","Data":"93fba251859722b1d7066ca5c39793f8157c0f803135597eeced9813aa1a113e"}
Oct 30 21:22:58.618213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.612768    4633 scope.go:122] "RemoveContainer" containerID="bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4"
Oct 30 21:22:58.618213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.613262    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"
Oct 30 21:22:58.648270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.648249    4633 scope.go:122] "RemoveContainer" containerID="bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4"
Oct 30 21:22:58.648593 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:22:58.648573    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4\": container with ID starting with bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4 not found: ID does not exist" containerID="bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4"
Oct 30 21:22:58.648662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.648606    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4"} err="failed to get container status \"bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4\": rpc error: code = NotFound desc = could not find container \"bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4\": container with ID starting with bbcd4bfe8f21ae9e8d7bc7875857f61e7e5ba79d110717beb078957d967152b4 not found: ID does not exist"
Oct 30 21:22:58.655474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.655448    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"]
Oct 30 21:22:58.661531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:58.661509    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4c6998-5cbf648dc-6sdfh"]
Oct 30 21:22:59.792953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:22:59.792921    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" path="/var/lib/kubelet/pods/e4001d79-f0a5-4dbe-bf55-d8ec626b2c40/volumes"
Oct 30 21:23:02.985003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:02.984961    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"]
Oct 30 21:23:02.985744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:02.985212    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc" podUID="450f246d-6b20-409e-8438-7d1ce7e777a6" containerName="payload-processing" containerID="cri-o://5eaff34d63ebea24df5bbd07edfe954fe31cfea2209c656323ab698f88884bff" gracePeriod=30
Oct 30 21:23:03.878723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:03.878638    4633 generic.go:356] "Generic (PLEG): container finished" podID="450f246d-6b20-409e-8438-7d1ce7e777a6" containerID="5eaff34d63ebea24df5bbd07edfe954fe31cfea2209c656323ab698f88884bff" exitCode=0
Oct 30 21:23:03.878991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:03.878963    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"]
Oct 30 21:23:03.879090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:03.879027    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc" event={"ID":"450f246d-6b20-409e-8438-7d1ce7e777a6","Type":"ContainerDied","Data":"5eaff34d63ebea24df5bbd07edfe954fe31cfea2209c656323ab698f88884bff"}
Oct 30 21:23:03.983405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:03.983369    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:23:04.076173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.076132    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450f246d-6b20-409e-8438-7d1ce7e777a6-kube-api-access-cgzmk\" (UniqueName: \"kubernetes.io/projected/450f246d-6b20-409e-8438-7d1ce7e777a6-kube-api-access-cgzmk\") pod \"450f246d-6b20-409e-8438-7d1ce7e777a6\" (UID: \"450f246d-6b20-409e-8438-7d1ce7e777a6\") "
Oct 30 21:23:04.076727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.076229    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450f246d-6b20-409e-8438-7d1ce7e777a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f246d-6b20-409e-8438-7d1ce7e777a6-plugins-config-volume\") pod \"450f246d-6b20-409e-8438-7d1ce7e777a6\" (UID: \"450f246d-6b20-409e-8438-7d1ce7e777a6\") "
Oct 30 21:23:04.077024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.077000    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450f246d-6b20-409e-8438-7d1ce7e777a6-plugins-config-volume" pod "450f246d-6b20-409e-8438-7d1ce7e777a6" (UID: "450f246d-6b20-409e-8438-7d1ce7e777a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:04.079349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.079305    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450f246d-6b20-409e-8438-7d1ce7e777a6-kube-api-access-cgzmk" pod "450f246d-6b20-409e-8438-7d1ce7e777a6" (UID: "450f246d-6b20-409e-8438-7d1ce7e777a6"). InnerVolumeSpecName "kube-api-access-cgzmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:04.178387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.178344    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgzmk\" (UniqueName: \"kubernetes.io/projected/450f246d-6b20-409e-8438-7d1ce7e777a6-kube-api-access-cgzmk\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.178387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.178388    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450f246d-6b20-409e-8438-7d1ce7e777a6-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.888187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.888155    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"
Oct 30 21:23:04.888371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.888187    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc" event={"ID":"450f246d-6b20-409e-8438-7d1ce7e777a6","Type":"ContainerDied","Data":"0d2200028b828311ff019723363b917649e8946b6376af1d0ab6bbdaf80cc6de"}
Oct 30 21:23:04.888371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.888251    4633 scope.go:122] "RemoveContainer" containerID="5eaff34d63ebea24df5bbd07edfe954fe31cfea2209c656323ab698f88884bff"
Oct 30 21:23:04.938130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.938055    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"]
Oct 30 21:23:04.944772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:04.944743    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-57d75b658b-mjttc"]
Oct 30 21:23:05.793056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:05.793023    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450f246d-6b20-409e-8438-7d1ce7e777a6" path="/var/lib/kubelet/pods/450f246d-6b20-409e-8438-7d1ce7e777a6/volumes"
Oct 30 21:23:09.060699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.052572    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"]
Oct 30 21:23:09.060699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.052840    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q" podUID="958f15d6-c86f-41f5-9f21-d7774f585966" containerName="maas-api" containerID="cri-o://88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78" gracePeriod=30
Oct 30 21:23:09.355033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.355006    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:23:09.533697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.532899    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/958f15d6-c86f-41f5-9f21-d7774f585966-kube-api-access-rxsmr\" (UniqueName: \"kubernetes.io/projected/958f15d6-c86f-41f5-9f21-d7774f585966-kube-api-access-rxsmr\") pod \"958f15d6-c86f-41f5-9f21-d7774f585966\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") "
Oct 30 21:23:09.533697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.533131    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-metrics-certs\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-metrics-certs\") pod \"958f15d6-c86f-41f5-9f21-d7774f585966\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") "
Oct 30 21:23:09.533697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.533167    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls\") pod \"958f15d6-c86f-41f5-9f21-d7774f585966\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") "
Oct 30 21:23:09.533697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.533271    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/958f15d6-c86f-41f5-9f21-d7774f585966-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958f15d6-c86f-41f5-9f21-d7774f585966-openshift-service-ca\") pod \"958f15d6-c86f-41f5-9f21-d7774f585966\" (UID: \"958f15d6-c86f-41f5-9f21-d7774f585966\") "
Oct 30 21:23:09.594093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.594015    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958f15d6-c86f-41f5-9f21-d7774f585966-openshift-service-ca" pod "958f15d6-c86f-41f5-9f21-d7774f585966" (UID: "958f15d6-c86f-41f5-9f21-d7774f585966"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:09.595515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.595464    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls" pod "958f15d6-c86f-41f5-9f21-d7774f585966" (UID: "958f15d6-c86f-41f5-9f21-d7774f585966"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:09.597316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.595982    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-metrics-certs" pod "958f15d6-c86f-41f5-9f21-d7774f585966" (UID: "958f15d6-c86f-41f5-9f21-d7774f585966"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:09.630562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.630519    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958f15d6-c86f-41f5-9f21-d7774f585966-kube-api-access-rxsmr" pod "958f15d6-c86f-41f5-9f21-d7774f585966" (UID: "958f15d6-c86f-41f5-9f21-d7774f585966"). InnerVolumeSpecName "kube-api-access-rxsmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:09.640453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.640398    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxsmr\" (UniqueName: \"kubernetes.io/projected/958f15d6-c86f-41f5-9f21-d7774f585966-kube-api-access-rxsmr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:09.640453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.640445    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:09.640743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.640462    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/958f15d6-c86f-41f5-9f21-d7774f585966-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:09.640743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.640476    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/958f15d6-c86f-41f5-9f21-d7774f585966-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:09.930837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.930797    4633 generic.go:356] "Generic (PLEG): container finished" podID="958f15d6-c86f-41f5-9f21-d7774f585966" containerID="88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78" exitCode=0
Oct 30 21:23:09.931040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.930849    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"
Oct 30 21:23:09.931040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.930885    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q" event={"ID":"958f15d6-c86f-41f5-9f21-d7774f585966","Type":"ContainerDied","Data":"88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78"}
Oct 30 21:23:09.931040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.930947    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q" event={"ID":"958f15d6-c86f-41f5-9f21-d7774f585966","Type":"ContainerDied","Data":"dca122e5925d096f3da8e9bb6a20aa04129156fe0e3ffe105bf3098888b7445c"}
Oct 30 21:23:09.931040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.930973    4633 scope.go:122] "RemoveContainer" containerID="88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78"
Oct 30 21:23:09.943023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.942998    4633 scope.go:122] "RemoveContainer" containerID="88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78"
Oct 30 21:23:09.943359 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:09.943322    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78\": container with ID starting with 88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78 not found: ID does not exist" containerID="88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78"
Oct 30 21:23:09.943430 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.943366    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78"} err="failed to get container status \"88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78\": rpc error: code = NotFound desc = could not find container \"88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78\": container with ID starting with 88b4a373ff5409b60cb71f9cab967af6742670a53b70d7d2ea0b68316a9d8e78 not found: ID does not exist"
Oct 30 21:23:09.970809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.970775    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"]
Oct 30 21:23:09.974908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:09.974882    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d87b5f-666cdc86c4-f8z2q"]
Oct 30 21:23:10.261480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.261388    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"]
Oct 30 21:23:10.272115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.261984    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28" podUID="bcb4e6ed-33c7-4841-bbf6-9c34211431bf" containerName="maas-api" containerID="cri-o://37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee" gracePeriod=30
Oct 30 21:23:10.796562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.796182    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:23:10.907721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.897584    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-kube-api-access-gwbhr\" (UniqueName: \"kubernetes.io/projected/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-kube-api-access-gwbhr\") pod \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") "
Oct 30 21:23:10.907721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.897788    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-metrics-certs\") pod \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") "
Oct 30 21:23:10.907721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.897822    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-openshift-service-ca\") pod \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") "
Oct 30 21:23:10.907721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.898019    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls\") pod \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\" (UID: \"bcb4e6ed-33c7-4841-bbf6-9c34211431bf\") "
Oct 30 21:23:10.908934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.908723    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-kube-api-access-gwbhr" pod "bcb4e6ed-33c7-4841-bbf6-9c34211431bf" (UID: "bcb4e6ed-33c7-4841-bbf6-9c34211431bf"). InnerVolumeSpecName "kube-api-access-gwbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:10.908934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.908877    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-openshift-service-ca" pod "bcb4e6ed-33c7-4841-bbf6-9c34211431bf" (UID: "bcb4e6ed-33c7-4841-bbf6-9c34211431bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:10.940713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.939141    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-metrics-certs" pod "bcb4e6ed-33c7-4841-bbf6-9c34211431bf" (UID: "bcb4e6ed-33c7-4841-bbf6-9c34211431bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:10.940713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.939272    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls" pod "bcb4e6ed-33c7-4841-bbf6-9c34211431bf" (UID: "bcb4e6ed-33c7-4841-bbf6-9c34211431bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:10.976733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.976293    4633 generic.go:356] "Generic (PLEG): container finished" podID="bcb4e6ed-33c7-4841-bbf6-9c34211431bf" containerID="37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee" exitCode=0
Oct 30 21:23:10.976733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.976378    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28" event={"ID":"bcb4e6ed-33c7-4841-bbf6-9c34211431bf","Type":"ContainerDied","Data":"37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee"}
Oct 30 21:23:10.976733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.976432    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28" event={"ID":"bcb4e6ed-33c7-4841-bbf6-9c34211431bf","Type":"ContainerDied","Data":"1e4e52c36f05857b39d637f9dd71f2fdf22e2f54cddc0b751aa2de1609aff3d4"}
Oct 30 21:23:10.976733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.976466    4633 scope.go:122] "RemoveContainer" containerID="37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee"
Oct 30 21:23:10.976733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:10.976655    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"
Oct 30 21:23:11.000216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.000112    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwbhr\" (UniqueName: \"kubernetes.io/projected/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-kube-api-access-gwbhr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:11.000216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.000150    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:11.000216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.000166    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:11.000216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.000179    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4e6ed-33c7-4841-bbf6-9c34211431bf-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:11.002955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.002853    4633 scope.go:122] "RemoveContainer" containerID="37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee"
Oct 30 21:23:11.003493 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:11.003450    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee\": container with ID starting with 37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee not found: ID does not exist" containerID="37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee"
Oct 30 21:23:11.003689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.003632    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee"} err="failed to get container status \"37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee\": rpc error: code = NotFound desc = could not find container \"37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee\": container with ID starting with 37b9ac55ce1bcbe5bb1f071031140247bd24e180d08ea756e3f71ff39c3b21ee not found: ID does not exist"
Oct 30 21:23:11.230654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.230550    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"]
Oct 30 21:23:11.242991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.242956    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-df078c-b847cd48b-lpv28"]
Oct 30 21:23:11.685821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.685771    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"]
Oct 30 21:23:11.686436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.686401    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy" containerID="cri-o://3f2d8a678144a63954cb386bee70cd64caa7ad9c7a053232efb40594f2fd865e" gracePeriod=30
Oct 30 21:23:11.869714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.868502    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="958f15d6-c86f-41f5-9f21-d7774f585966" path="/var/lib/kubelet/pods/958f15d6-c86f-41f5-9f21-d7774f585966/volumes"
Oct 30 21:23:11.887692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:11.884760    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb4e6ed-33c7-4841-bbf6-9c34211431bf" path="/var/lib/kubelet/pods/bcb4e6ed-33c7-4841-bbf6-9c34211431bf/volumes"
Oct 30 21:23:14.090881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:14.090778    4633 generic.go:356] "Generic (PLEG): container finished" podID="c29883ff-2844-4d5f-b016-9e87407390fc" containerID="441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650" exitCode=0
Oct 30 21:23:14.091413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:14.090993    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" event={"ID":"c29883ff-2844-4d5f-b016-9e87407390fc","Type":"ContainerDied","Data":"441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650"}
Oct 30 21:23:14.091522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:14.091500    4633 scope.go:122] "RemoveContainer" containerID="441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650"
Oct 30 21:23:14.094301 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:14.094256    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:14.094432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:14.094352    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:14.094432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:14.094402    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:15.110873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:15.110837    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" event={"ID":"c29883ff-2844-4d5f-b016-9e87407390fc","Type":"ContainerStarted","Data":"4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c"}
Oct 30 21:23:15.138706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:15.137096    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:23:15.139432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:15.138891    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:23:15.139432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:15.138980    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:23:16.138925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:16.138892    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:23:16.139439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:16.138971    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:23:17.164385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.141353    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:23:17.164385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.141427    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:23:17.356455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.356422    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"]
Oct 30 21:23:17.356865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.356840    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" containerID="cri-o://ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675" gracePeriod=30
Oct 30 21:23:17.359300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.358817    4633 generic.go:356] "Generic (PLEG): container finished" podID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerID="3f2d8a678144a63954cb386bee70cd64caa7ad9c7a053232efb40594f2fd865e" exitCode=0
Oct 30 21:23:17.359300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.359125    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" event={"ID":"730d60c8-aa12-4e39-9fb7-c43aeb8aa656","Type":"ContainerDied","Data":"3f2d8a678144a63954cb386bee70cd64caa7ad9c7a053232efb40594f2fd865e"}
Oct 30 21:23:17.359300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.359177    4633 scope.go:122] "RemoveContainer" containerID="54a06d4426560ba0962609f24997fb522a0becf6e03c89a1a341e35be1d90084"
Oct 30 21:23:17.379319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.379289    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:23:17.494713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.488750    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-certs\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.494713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.489143    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istiod-ca-cert\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.494713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.489452    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-token\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-token\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.494713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.493816    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-certs" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:17.494713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.494399    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-data\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.511994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.502963    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-data" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:17.511994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.511598    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istiod-ca-cert" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:17.517697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.513127    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-kube-api-access-2b4g9\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-kube-api-access-2b4g9\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.517697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.513242    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-socket\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.517697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.513290    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-credential-socket\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.517697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.513317    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-envoy\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.517697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.513383    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-podinfo\") pod \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\" (UID: \"730d60c8-aa12-4e39-9fb7-c43aeb8aa656\") "
Oct 30 21:23:17.517697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.514203    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.517697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.514226    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.517697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.514240    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.527703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.520279    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-credential-socket" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:17.527703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.520608    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-socket" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:17.558696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.557038    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-token" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:17.558696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.557193    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-kube-api-access-2b4g9" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "kube-api-access-2b4g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:17.558696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.557325    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-podinfo" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:17.567755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.566834    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-envoy" pod "730d60c8-aa12-4e39-9fb7-c43aeb8aa656" (UID: "730d60c8-aa12-4e39-9fb7-c43aeb8aa656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:17.628764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.616063    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b4g9\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-kube-api-access-2b4g9\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.629094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.629013    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.629292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.629275    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.629388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.629376    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.629478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.629467    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:17.629564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:17.629550    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730d60c8-aa12-4e39-9fb7-c43aeb8aa656-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:18.138817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.138770    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:23:18.139000 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.138858    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:23:18.380764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.380723    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf" event={"ID":"730d60c8-aa12-4e39-9fb7-c43aeb8aa656","Type":"ContainerDied","Data":"df4eeb2c0cd7d6cfb1073b10e3da767a29144ba9c62c33007cc6827aa86eeb25"}
Oct 30 21:23:18.381188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.380798    4633 scope.go:122] "RemoveContainer" containerID="3f2d8a678144a63954cb386bee70cd64caa7ad9c7a053232efb40594f2fd865e"
Oct 30 21:23:18.381188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.380736    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"
Oct 30 21:23:18.441694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.441314    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"]
Oct 30 21:23:18.446933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.446905    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-09eb50-openshift-default-79ddfb8859-wmqqf"]
Oct 30 21:23:18.658108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.658069    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"]
Oct 30 21:23:18.659436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659409    4633 cpu_manager.go:414] "containerMap: removing container" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy"
Oct 30 21:23:18.659436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659437    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy"
Oct 30 21:23:18.659598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659457    4633 cpu_manager.go:414] "containerMap: removing container" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy"
Oct 30 21:23:18.659598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659465    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy"
Oct 30 21:23:18.659598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659520    4633 cpu_manager.go:414] "containerMap: removing container" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy"
Oct 30 21:23:18.659598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659530    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy"
Oct 30 21:23:18.659598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659556    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy"
Oct 30 21:23:18.659598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659563    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy"
Oct 30 21:23:18.659598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659576    4633 cpu_manager.go:414] "containerMap: removing container" podUID="958f15d6-c86f-41f5-9f21-d7774f585966" containerName="maas-api"
Oct 30 21:23:18.659598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659583    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="958f15d6-c86f-41f5-9f21-d7774f585966" containerName="maas-api"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659606    4633 cpu_manager.go:414] "containerMap: removing container" podUID="bcb4e6ed-33c7-4841-bbf6-9c34211431bf" containerName="maas-api"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659614    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb4e6ed-33c7-4841-bbf6-9c34211431bf" containerName="maas-api"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659636    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" containerName="maas-api"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659644    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" containerName="maas-api"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659653    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659659    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659690    4633 cpu_manager.go:414] "containerMap: removing container" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659700    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659713    4633 cpu_manager.go:414] "containerMap: removing container" podUID="450f246d-6b20-409e-8438-7d1ce7e777a6" containerName="payload-processing"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659720    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450f246d-6b20-409e-8438-7d1ce7e777a6" containerName="payload-processing"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659895    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy"
Oct 30 21:23:18.659906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659909    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy"
Oct 30 21:23:18.660246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659920    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="958f15d6-c86f-41f5-9f21-d7774f585966" containerName="maas-api"
Oct 30 21:23:18.660246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659932    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb4e6ed-33c7-4841-bbf6-9c34211431bf" containerName="maas-api"
Oct 30 21:23:18.660246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659944    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4001d79-f0a5-4dbe-bf55-d8ec626b2c40" containerName="maas-api"
Oct 30 21:23:18.660246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659953    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a2699d-77ea-4e27-ac72-8428c096e798" containerName="istio-proxy"
Oct 30 21:23:18.660246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659964    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="450f246d-6b20-409e-8438-7d1ce7e777a6" containerName="payload-processing"
Oct 30 21:23:18.660246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659976    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy"
Oct 30 21:23:18.660246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.659987    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" containerName="istio-proxy"
Oct 30 21:23:18.665656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.665638    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.673076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.672356    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0fc028-openshift-default-dockercfg-f89m5\""
Oct 30 21:23:18.675154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.675118    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"]
Oct 30 21:23:18.754147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754024    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-podinfo\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.754147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754083    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-socket\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.754147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754117    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-certs\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.754446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754207    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-credential-socket\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.754446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754265    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-envoy\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.754446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754306    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istiod-ca-cert\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.754608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754510    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-data\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.754663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754637    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-token\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.754794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.754766    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrb88\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-kube-api-access-zrb88\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.868828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.868788    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-data\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.869081    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-token\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.869141    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrb88\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-kube-api-access-zrb88\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.869244    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-podinfo\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.869284    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-socket\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.869318    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-certs\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.869413    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-credential-socket\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.869474    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-envoy\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.869519    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istiod-ca-cert\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.870354    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istiod-ca-cert\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.871558    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-data\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.874335    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-socket\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.893696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.881196    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-podinfo\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.897481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.896259    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-credential-socket\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.897481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.896499    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-certs\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.950743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.929177    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-envoy\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.950743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.930857    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn" podUID="2d19f591-0fab-472f-8a17-4bd876fb6fc5" containerName="main" containerID="cri-o://4827b2cedbe1f210eb3036372d6e0139be2bdd49ac03f8e7f0bd277331398b77" gracePeriod=45
Oct 30 21:23:18.952527 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.952497    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrb88\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-kube-api-access-zrb88\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:18.986829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.984946    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-token\") pod \"e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") " pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:19.003414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:18.994971    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:19.076694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.070060    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn" podUID="2d19f591-0fab-472f-8a17-4bd876fb6fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": read tcp 10.133.0.2:54446->10.133.0.140:8000: read: connection reset by peer"
Oct 30 21:23:19.140698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.139108    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:23:19.140698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.139175    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:23:19.510371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.510260    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"]
Oct 30 21:23:19.549696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.542813    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"]
Oct 30 21:23:19.549696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.543132    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy" containerID="cri-o://37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95" gracePeriod=30
Oct 30 21:23:19.561815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.561751    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:19.561968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.561888    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:19.562040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.561969    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:19.563584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.563560    4633 generic.go:356] "Generic (PLEG): container finished" podID="2d19f591-0fab-472f-8a17-4bd876fb6fc5" containerID="4827b2cedbe1f210eb3036372d6e0139be2bdd49ac03f8e7f0bd277331398b77" exitCode=0
Oct 30 21:23:19.563954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.563931    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn" event={"ID":"2d19f591-0fab-472f-8a17-4bd876fb6fc5","Type":"ContainerDied","Data":"4827b2cedbe1f210eb3036372d6e0139be2bdd49ac03f8e7f0bd277331398b77"}
Oct 30 21:23:19.579356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.579327    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:23:19.624449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.624411    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d19f591-0fab-472f-8a17-4bd876fb6fc5-kube-api-access-j9hr5\" (UniqueName: \"kubernetes.io/projected/2d19f591-0fab-472f-8a17-4bd876fb6fc5-kube-api-access-j9hr5\") pod \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") "
Oct 30 21:23:19.626986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.626959    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-dshm\") pod \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") "
Oct 30 21:23:19.627540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.627522    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tls-certs\") pod \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") "
Oct 30 21:23:19.628098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.628080    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-home\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-home\") pod \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") "
Oct 30 21:23:19.632858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.632838    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tmp-dir\") pod \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") "
Oct 30 21:23:19.633288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.633269    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-model-cache\") pod \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\" (UID: \"2d19f591-0fab-472f-8a17-4bd876fb6fc5\") "
Oct 30 21:23:19.634411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.632749    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-home" pod "2d19f591-0fab-472f-8a17-4bd876fb6fc5" (UID: "2d19f591-0fab-472f-8a17-4bd876fb6fc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:19.634540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.633197    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tmp-dir" pod "2d19f591-0fab-472f-8a17-4bd876fb6fc5" (UID: "2d19f591-0fab-472f-8a17-4bd876fb6fc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:19.634646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.633569    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-model-cache" pod "2d19f591-0fab-472f-8a17-4bd876fb6fc5" (UID: "2d19f591-0fab-472f-8a17-4bd876fb6fc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:19.667426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.667388    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-dshm" pod "2d19f591-0fab-472f-8a17-4bd876fb6fc5" (UID: "2d19f591-0fab-472f-8a17-4bd876fb6fc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:19.668900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.668871    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d19f591-0fab-472f-8a17-4bd876fb6fc5-kube-api-access-j9hr5" pod "2d19f591-0fab-472f-8a17-4bd876fb6fc5" (UID: "2d19f591-0fab-472f-8a17-4bd876fb6fc5"). InnerVolumeSpecName "kube-api-access-j9hr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:19.697884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.697847    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tls-certs" pod "2d19f591-0fab-472f-8a17-4bd876fb6fc5" (UID: "2d19f591-0fab-472f-8a17-4bd876fb6fc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:19.814963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.802643    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.814963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.802697    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.814963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.802711    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.814963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.802727    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9hr5\" (UniqueName: \"kubernetes.io/projected/2d19f591-0fab-472f-8a17-4bd876fb6fc5-kube-api-access-j9hr5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.814963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.802742    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19f591-0fab-472f-8a17-4bd876fb6fc5-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.814963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.802755    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19f591-0fab-472f-8a17-4bd876fb6fc5-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:19.902922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:19.902743    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730d60c8-aa12-4e39-9fb7-c43aeb8aa656" path="/var/lib/kubelet/pods/730d60c8-aa12-4e39-9fb7-c43aeb8aa656/volumes"
Oct 30 21:23:20.138742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.138092    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:23:20.138742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.138153    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:23:20.699526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.699175    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" event={"ID":"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b","Type":"ContainerStarted","Data":"60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19"}
Oct 30 21:23:20.699526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.699246    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" event={"ID":"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b","Type":"ContainerStarted","Data":"922b9b9b66bf9263dbe1242fdc161a00b530ee71e8aead81f8db25b5b811a4ff"}
Oct 30 21:23:20.701108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.701084    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn" event={"ID":"2d19f591-0fab-472f-8a17-4bd876fb6fc5","Type":"ContainerDied","Data":"e18e618f4cfd7ce4184a73796080039eacb3187682a0fff860e291e06a0fd24a"}
Oct 30 21:23:20.701234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.701125    4633 scope.go:122] "RemoveContainer" containerID="4827b2cedbe1f210eb3036372d6e0139be2bdd49ac03f8e7f0bd277331398b77"
Oct 30 21:23:20.701300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.701265    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"
Oct 30 21:23:20.742919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.742857    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" podStartSLOduration=2.742836282 podStartE2EDuration="2.742836282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:20.73025627 +0000 UTC m=+1309.079983668" watchObservedRunningTime="2026-10-30 21:23:20.742836282 +0000 UTC m=+1309.092563679"
Oct 30 21:23:20.752123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.752096    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"]
Oct 30 21:23:20.758981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:20.758960    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4c6998/ipp-route-4c6998-kserve-65d9fffbb9-bx8dn"]
Oct 30 21:23:21.000890 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.000803    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:21.006995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.006972    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:21.136968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.136928    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 21:23:21.137148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.137018    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 21:23:21.526054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.526021    4633 patch_prober.go:29] interesting pod/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:23:21.526362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.526336    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:23:21.710230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.710196    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:21.711518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.711496    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:23:21.793185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:21.793110    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d19f591-0fab-472f-8a17-4bd876fb6fc5" path="/var/lib/kubelet/pods/2d19f591-0fab-472f-8a17-4bd876fb6fc5/volumes"
Oct 30 21:23:22.142426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.142334    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:23:22.142837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.142819    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:23:22.143569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.143548    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:23:22.634137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.634110    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:23:22.721536 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.721497    4633 generic.go:356] "Generic (PLEG): container finished" podID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerID="ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675" exitCode=0
Oct 30 21:23:22.722536 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.722513    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"
Oct 30 21:23:22.722917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.722896    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" event={"ID":"46ce198d-a5bb-4115-bffd-206c9faf4629","Type":"ContainerDied","Data":"ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675"}
Oct 30 21:23:22.722980 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.722947    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk" event={"ID":"46ce198d-a5bb-4115-bffd-206c9faf4629","Type":"ContainerDied","Data":"faf23fe0d8f6689f2251e9b2e6f162ab26a9df7ba6462e0ceb15b41c37b1ca3a"}
Oct 30 21:23:22.723019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.722982    4633 scope.go:122] "RemoveContainer" containerID="ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675"
Oct 30 21:23:22.725971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.725949    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-token\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-token\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.726103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.726024    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-socket\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.726183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.726146    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-envoy\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.726336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.726301    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-socket" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:22.726537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.726513    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-credential-socket\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.726715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.726702    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-kube-api-access-mwjnn\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-kube-api-access-mwjnn\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.726784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.726737    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-certs\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.726784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.726776    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-data\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.727073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.726981    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-credential-socket" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:22.727199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.727176    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-data" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:22.727252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.727240    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-certs" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:22.727389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.727372    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ce198d-a5bb-4115-bffd-206c9faf4629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ce198d-a5bb-4115-bffd-206c9faf4629-istiod-ca-cert\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.727469 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.727436    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-podinfo\") pod \"46ce198d-a5bb-4115-bffd-206c9faf4629\" (UID: \"46ce198d-a5bb-4115-bffd-206c9faf4629\") "
Oct 30 21:23:22.728436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.727785    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ce198d-a5bb-4115-bffd-206c9faf4629-istiod-ca-cert" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:22.728540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.728462    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-token" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:22.728685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.728650    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.728798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.728778    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.728856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.728803    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.728856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.728818    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.728856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.728831    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ce198d-a5bb-4115-bffd-206c9faf4629-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.728856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.728843    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.728856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.728748    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-envoy" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:22.730563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.730533    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-podinfo" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:22.731659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.731634    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-kube-api-access-mwjnn" pod "46ce198d-a5bb-4115-bffd-206c9faf4629" (UID: "46ce198d-a5bb-4115-bffd-206c9faf4629"). InnerVolumeSpecName "kube-api-access-mwjnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:22.758990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.758967    4633 scope.go:122] "RemoveContainer" containerID="47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11"
Oct 30 21:23:22.769435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.769408    4633 scope.go:122] "RemoveContainer" containerID="ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675"
Oct 30 21:23:22.769754 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:22.769736    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675\": container with ID starting with ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675 not found: ID does not exist" containerID="ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675"
Oct 30 21:23:22.769811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.769765    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675"} err="failed to get container status \"ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675\": rpc error: code = NotFound desc = could not find container \"ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675\": container with ID starting with ea6ee7f05e6bf268a30ad644b3962e6bf9432d6ea32d8adf252a56882e374675 not found: ID does not exist"
Oct 30 21:23:22.769811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.769783    4633 scope.go:122] "RemoveContainer" containerID="47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11"
Oct 30 21:23:22.770091 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:22.770068    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11\": container with ID starting with 47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11 not found: ID does not exist" containerID="47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11"
Oct 30 21:23:22.770154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.770099    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11"} err="failed to get container status \"47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11\": rpc error: code = NotFound desc = could not find container \"47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11\": container with ID starting with 47ef3596b280c6a045fd5eef6460b0dc2fa35ba0bb3e403e89e5599bf041ac11 not found: ID does not exist"
Oct 30 21:23:22.830916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.830882    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjnn\" (UniqueName: \"kubernetes.io/projected/46ce198d-a5bb-4115-bffd-206c9faf4629-kube-api-access-mwjnn\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.830916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.830914    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:22.830916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:22.830925    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ce198d-a5bb-4115-bffd-206c9faf4629-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:23.052988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:23.052948    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"]
Oct 30 21:23:23.058498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:23.058471    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d87b5f-openshift-default-5c55d4849f-fgxxk"]
Oct 30 21:23:23.792775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:23.792741    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" path="/var/lib/kubelet/pods/46ce198d-a5bb-4115-bffd-206c9faf4629/volumes"
Oct 30 21:23:24.388009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.387976    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"]
Oct 30 21:23:24.388654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388634    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2d19f591-0fab-472f-8a17-4bd876fb6fc5" containerName="main"
Oct 30 21:23:24.388654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388650    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d19f591-0fab-472f-8a17-4bd876fb6fc5" containerName="main"
Oct 30 21:23:24.388785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388664    4633 cpu_manager.go:414] "containerMap: removing container" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy"
Oct 30 21:23:24.388785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388687    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy"
Oct 30 21:23:24.388785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388704    4633 cpu_manager.go:414] "containerMap: removing container" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy"
Oct 30 21:23:24.388785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388709    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy"
Oct 30 21:23:24.388903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388800    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d19f591-0fab-472f-8a17-4bd876fb6fc5" containerName="main"
Oct 30 21:23:24.388903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388808    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy"
Oct 30 21:23:24.388903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388817    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ce198d-a5bb-4115-bffd-206c9faf4629" containerName="istio-proxy"
Oct 30 21:23:24.388903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.388824    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01d8427-ce69-4098-b6c9-cf2eb7530091" containerName="istio-proxy"
Oct 30 21:23:24.393829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.393813    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:24.398761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.398734    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0fc028\""
Oct 30 21:23:24.398934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.398812    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0fc028-dockercfg-47nhf\""
Oct 30 21:23:24.404167 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.404145    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"]
Oct 30 21:23:24.434734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.434697    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"]
Oct 30 21:23:24.439244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.439221    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:24.446821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.446783    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"]
Oct 30 21:23:24.454969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.454940    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df92x\" (UniqueName: \"kubernetes.io/projected/5f516526-0c1d-4adf-8500-75aa668e2f1e-kube-api-access-df92x\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn\" (UID: \"5f516526-0c1d-4adf-8500-75aa668e2f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:24.455148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.455109    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f516526-0c1d-4adf-8500-75aa668e2f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn\" (UID: \"5f516526-0c1d-4adf-8500-75aa668e2f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:24.567700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.561837    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f516526-0c1d-4adf-8500-75aa668e2f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn\" (UID: \"5f516526-0c1d-4adf-8500-75aa668e2f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:24.567700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.561930    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1121df-e361-4e6b-8953-05c737617b9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s\" (UID: \"fd1121df-e361-4e6b-8953-05c737617b9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:24.567700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.562064    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df92x\" (UniqueName: \"kubernetes.io/projected/5f516526-0c1d-4adf-8500-75aa668e2f1e-kube-api-access-df92x\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn\" (UID: \"5f516526-0c1d-4adf-8500-75aa668e2f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:24.567700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.563385    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f516526-0c1d-4adf-8500-75aa668e2f1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn\" (UID: \"5f516526-0c1d-4adf-8500-75aa668e2f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:24.687042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.666728    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1121df-e361-4e6b-8953-05c737617b9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s\" (UID: \"fd1121df-e361-4e6b-8953-05c737617b9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:24.687042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.666893    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6mn\" (UniqueName: \"kubernetes.io/projected/fd1121df-e361-4e6b-8953-05c737617b9e-kube-api-access-wk6mn\") pod \"payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s\" (UID: \"fd1121df-e361-4e6b-8953-05c737617b9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:24.687042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.668032    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1121df-e361-4e6b-8953-05c737617b9e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s\" (UID: \"fd1121df-e361-4e6b-8953-05c737617b9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:24.701709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.699762    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df92x\" (UniqueName: \"kubernetes.io/projected/5f516526-0c1d-4adf-8500-75aa668e2f1e-kube-api-access-df92x\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn\" (UID: \"5f516526-0c1d-4adf-8500-75aa668e2f1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:24.741694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.725749    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:24.770425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.768167    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6mn\" (UniqueName: \"kubernetes.io/projected/fd1121df-e361-4e6b-8953-05c737617b9e-kube-api-access-wk6mn\") pod \"payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s\" (UID: \"fd1121df-e361-4e6b-8953-05c737617b9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:24.780395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.780367    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6mn\" (UniqueName: \"kubernetes.io/projected/fd1121df-e361-4e6b-8953-05c737617b9e-kube-api-access-wk6mn\") pod \"payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s\" (UID: \"fd1121df-e361-4e6b-8953-05c737617b9e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:24.989340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:24.989313    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:23:25.017786 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:25.017749    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:23:25.064397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.064358    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:25.142906 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:23:25.138000    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f516526_0c1d_4adf_8500_75aa668e2f1e.slice/crio-2a3b8b508a5e3bf8aa2e2bf6e66eaf4939784e3d03ca69f79e86f9b89b4d3921 WatchSource:0}: Error finding container 2a3b8b508a5e3bf8aa2e2bf6e66eaf4939784e3d03ca69f79e86f9b89b4d3921: Status 404 returned error can't find the container with id 2a3b8b508a5e3bf8aa2e2bf6e66eaf4939784e3d03ca69f79e86f9b89b4d3921
Oct 30 21:23:25.142906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.142364    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"]
Oct 30 21:23:25.186354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186319    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-podinfo\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.186526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186386    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-credential-socket\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.187544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186604    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-socket\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.187544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186649    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-data\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.187544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186760    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-certs\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.187544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186833    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-kube-api-access-59r4k\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-kube-api-access-59r4k\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.187544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186869    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-token\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-token\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.187544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186893    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-envoy\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.187544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.186977    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52aca4bf-6016-4c4b-8a5d-29aae1277413-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52aca4bf-6016-4c4b-8a5d-29aae1277413-istiod-ca-cert\") pod \"52aca4bf-6016-4c4b-8a5d-29aae1277413\" (UID: \"52aca4bf-6016-4c4b-8a5d-29aae1277413\") "
Oct 30 21:23:25.187544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.187366    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-certs" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:25.188088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.187828    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.188243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.188201    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-data" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:25.188384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.188333    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52aca4bf-6016-4c4b-8a5d-29aae1277413-istiod-ca-cert" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:25.188384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.188359    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-credential-socket" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:25.188506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.188420    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-socket" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:25.188844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.188824    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-podinfo" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:25.189939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.189917    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-envoy" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:25.190566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.190429    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-kube-api-access-59r4k" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "kube-api-access-59r4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:25.190566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.190507    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-token" pod "52aca4bf-6016-4c4b-8a5d-29aae1277413" (UID: "52aca4bf-6016-4c4b-8a5d-29aae1277413"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:25.289205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.289174    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.289205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.289202    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.289205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.289212    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59r4k\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-kube-api-access-59r4k\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.289441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.289223    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.289441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.289232    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.289441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.289241    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52aca4bf-6016-4c4b-8a5d-29aae1277413-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.289441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.289249    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52aca4bf-6016-4c4b-8a5d-29aae1277413-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.289441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.289257    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52aca4bf-6016-4c4b-8a5d-29aae1277413-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:25.431981 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:23:25.431953    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1121df_e361_4e6b_8953_05c737617b9e.slice/crio-2a39ebfc75b71a292b161d0cb7d6b1375d01d8be16422c4edf5fbc34b2f6fa70 WatchSource:0}: Error finding container 2a39ebfc75b71a292b161d0cb7d6b1375d01d8be16422c4edf5fbc34b2f6fa70: Status 404 returned error can't find the container with id 2a39ebfc75b71a292b161d0cb7d6b1375d01d8be16422c4edf5fbc34b2f6fa70
Oct 30 21:23:25.434182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.434161    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"]
Oct 30 21:23:25.835318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.835234    4633 generic.go:356] "Generic (PLEG): container finished" podID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerID="37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95" exitCode=0
Oct 30 21:23:25.835466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.835397    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"
Oct 30 21:23:25.838011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.837992    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:25.838150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.838036    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:25.838150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.838059    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn" event={"ID":"5f516526-0c1d-4adf-8500-75aa668e2f1e","Type":"ContainerStarted","Data":"05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95"}
Oct 30 21:23:25.838150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.838085    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn" event={"ID":"5f516526-0c1d-4adf-8500-75aa668e2f1e","Type":"ContainerStarted","Data":"2a3b8b508a5e3bf8aa2e2bf6e66eaf4939784e3d03ca69f79e86f9b89b4d3921"}
Oct 30 21:23:25.838150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.838105    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s" event={"ID":"fd1121df-e361-4e6b-8953-05c737617b9e","Type":"ContainerStarted","Data":"263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7"}
Oct 30 21:23:25.838150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.838125    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s" event={"ID":"fd1121df-e361-4e6b-8953-05c737617b9e","Type":"ContainerStarted","Data":"2a39ebfc75b71a292b161d0cb7d6b1375d01d8be16422c4edf5fbc34b2f6fa70"}
Oct 30 21:23:25.838150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.838144    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" event={"ID":"52aca4bf-6016-4c4b-8a5d-29aae1277413","Type":"ContainerDied","Data":"37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95"}
Oct 30 21:23:25.838421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.838174    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg" event={"ID":"52aca4bf-6016-4c4b-8a5d-29aae1277413","Type":"ContainerDied","Data":"583182509c72498546aeeacf3b42d13a246d14ca17d4996d9d782377322b19f1"}
Oct 30 21:23:25.838421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.838203    4633 scope.go:122] "RemoveContainer" containerID="37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95"
Oct 30 21:23:25.851487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.851462    4633 scope.go:122] "RemoveContainer" containerID="ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982"
Oct 30 21:23:25.852756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.852718    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn" podStartSLOduration=1.852704316 podStartE2EDuration="1.852704316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:25.847079376 +0000 UTC m=+1314.196806773" watchObservedRunningTime="2026-10-30 21:23:25.852704316 +0000 UTC m=+1314.202431714"
Oct 30 21:23:25.862552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.862527    4633 scope.go:122] "RemoveContainer" containerID="37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95"
Oct 30 21:23:25.862860 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:25.862842    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95\": container with ID starting with 37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95 not found: ID does not exist" containerID="37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95"
Oct 30 21:23:25.862920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.862868    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95"} err="failed to get container status \"37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95\": rpc error: code = NotFound desc = could not find container \"37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95\": container with ID starting with 37f5bd3077a2cca19f14562245aa080e9fa0ddb257288ee71a24e94f1a637a95 not found: ID does not exist"
Oct 30 21:23:25.862920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.862884    4633 scope.go:122] "RemoveContainer" containerID="ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982"
Oct 30 21:23:25.863133 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:25.863115    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982\": container with ID starting with ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982 not found: ID does not exist" containerID="ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982"
Oct 30 21:23:25.863183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.863138    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982"} err="failed to get container status \"ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982\": rpc error: code = NotFound desc = could not find container \"ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982\": container with ID starting with ab8c0329118ddc26ab829cba71edbfb45a50982ca5a4357cf09d3ce67f883982 not found: ID does not exist"
Oct 30 21:23:25.871955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.871911    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s" podStartSLOduration=1.871899647 podStartE2EDuration="1.871899647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:25.866361488 +0000 UTC m=+1314.216088884" watchObservedRunningTime="2026-10-30 21:23:25.871899647 +0000 UTC m=+1314.221627043"
Oct 30 21:23:25.881921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.881899    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"]
Oct 30 21:23:25.888307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:25.888284    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-df078c-openshift-default-78b97c767c-nfzjg"]
Oct 30 21:23:26.598230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:26.598196    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:23:27.794320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:27.794280    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" path="/var/lib/kubelet/pods/52aca4bf-6016-4c4b-8a5d-29aae1277413/volumes"
Oct 30 21:23:28.274948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.274899    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"]
Oct 30 21:23:28.276016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.275991    4633 cpu_manager.go:414] "containerMap: removing container" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy"
Oct 30 21:23:28.276016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.276013    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy"
Oct 30 21:23:28.276210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.276028    4633 cpu_manager.go:414] "containerMap: removing container" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy"
Oct 30 21:23:28.276210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.276036    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy"
Oct 30 21:23:28.276210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.276181    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy"
Oct 30 21:23:28.276210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.276191    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="52aca4bf-6016-4c4b-8a5d-29aae1277413" containerName="istio-proxy"
Oct 30 21:23:28.281991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.281971    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.304660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.304631    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-83921a-serving-cert\""
Oct 30 21:23:28.314600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.314247    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"]
Oct 30 21:23:28.383608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.383548    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7625v\" (UniqueName: \"kubernetes.io/projected/3a933c8c-6095-426d-aed1-0bdfcc92aa15-kube-api-access-7625v\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.383823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.383776    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-metrics-certs\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.383900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.383822    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a933c8c-6095-426d-aed1-0bdfcc92aa15-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.383971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.383945    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-maas-api-tls\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.509728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.508482    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-metrics-certs\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.509728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.509101    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a933c8c-6095-426d-aed1-0bdfcc92aa15-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.517530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.517493    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a933c8c-6095-426d-aed1-0bdfcc92aa15-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.518060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.518034    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-maas-api-tls\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.518947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.518926    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7625v\" (UniqueName: \"kubernetes.io/projected/3a933c8c-6095-426d-aed1-0bdfcc92aa15-kube-api-access-7625v\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.524586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.524526    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-metrics-certs\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.531808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.528380    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-maas-api-tls\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.535696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.532038    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7625v\" (UniqueName: \"kubernetes.io/projected/3a933c8c-6095-426d-aed1-0bdfcc92aa15-kube-api-access-7625v\") pod \"maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.628378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.628347    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:28.982964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:28.982929    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"]
Oct 30 21:23:29.961048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:29.961013    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp" event={"ID":"3a933c8c-6095-426d-aed1-0bdfcc92aa15","Type":"ContainerStarted","Data":"c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87"}
Oct 30 21:23:29.961236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:29.961075    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:29.961236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:29.961095    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp" event={"ID":"3a933c8c-6095-426d-aed1-0bdfcc92aa15","Type":"ContainerStarted","Data":"882137a54fa9d41921f37497dba75c68051910ea9f109df85bc72845f42aff94"}
Oct 30 21:23:29.980691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:29.980617    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp" podStartSLOduration=1.569023888 podStartE2EDuration="1.980604959s" totalImagesPullingTime="411.581071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:29.978741366 +0000 UTC m=+1318.328468762" watchObservedRunningTime="2026-10-30 21:23:29.980604959 +0000 UTC m=+1318.330332354"
Oct 30 21:23:32.263720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.263639    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"]
Oct 30 21:23:32.287476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.287429    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.301534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.301504    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0fc028-serving-cert\""
Oct 30 21:23:32.309099 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.308363    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"]
Oct 30 21:23:32.416593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.416547    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmsx\" (UniqueName: \"kubernetes.io/projected/2c360e20-b181-404b-988c-ea360ee107e9-kube-api-access-pxmsx\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.416593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.416591    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c360e20-b181-404b-988c-ea360ee107e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.416853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.416825    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.416938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.416926    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.524748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.520798    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.524748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.520932    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.524748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.521129    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmsx\" (UniqueName: \"kubernetes.io/projected/2c360e20-b181-404b-988c-ea360ee107e9-kube-api-access-pxmsx\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.524748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.521160    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c360e20-b181-404b-988c-ea360ee107e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.524748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.522062    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c360e20-b181-404b-988c-ea360ee107e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.539695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.539537    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmsx\" (UniqueName: \"kubernetes.io/projected/2c360e20-b181-404b-988c-ea360ee107e9-kube-api-access-pxmsx\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.542753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.542309    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.545367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.544578    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-metrics-certs\") pod \"maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.617651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.617617    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:32.940438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:32.940408    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"]
Oct 30 21:23:33.016209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:33.016177    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg" event={"ID":"2c360e20-b181-404b-988c-ea360ee107e9","Type":"ContainerStarted","Data":"dfce9b6e741345b8811e1bdbc108fc5447fa15fa0d08e9c526716d35c2157f59"}
Oct 30 21:23:34.029930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:34.029898    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg" event={"ID":"2c360e20-b181-404b-988c-ea360ee107e9","Type":"ContainerStarted","Data":"707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da"}
Oct 30 21:23:34.030435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:34.030018    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:34.054206 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:34.053483    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg" podStartSLOduration=1.658470925 podStartE2EDuration="2.053464941s" totalImagesPullingTime="394.994016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:34.049527459 +0000 UTC m=+1322.399254856" watchObservedRunningTime="2026-10-30 21:23:34.053464941 +0000 UTC m=+1322.403192339"
Oct 30 21:23:36.006955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:36.006924    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:23:36.872201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:36.872174    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:37.882862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:37.882834    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:40.043728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:40.043702    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:23:43.250921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:43.250889    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"]
Oct 30 21:23:43.261904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:43.261861    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"]
Oct 30 21:23:45.436118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.436084    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"]
Oct 30 21:23:45.442245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.442214    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.455081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.455048    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"]
Oct 30 21:23:45.461481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.461447    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-18b363-openshift-default-dockercfg-k8jcw\""
Oct 30 21:23:45.623696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.582578    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-envoy\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.637517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.637485    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-istio-token\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.639210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.639166    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10996acb-8603-4451-804e-d808664017d5-istio-podinfo\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.639441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.639425    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-data\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.639824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.639805    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10996acb-8603-4451-804e-d808664017d5-istiod-ca-cert\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.640012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.639998    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-certs\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.640180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.640165    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-socket\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.640323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.640311    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-credential-socket\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.640487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.640474    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gcz\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-kube-api-access-g4gcz\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.748594    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-istio-token\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.748703    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10996acb-8603-4451-804e-d808664017d5-istio-podinfo\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.748741    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-data\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.748967    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10996acb-8603-4451-804e-d808664017d5-istiod-ca-cert\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.749011    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-certs\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.749067    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-socket\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.749091    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-credential-socket\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.749136    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gcz\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-kube-api-access-g4gcz\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.753756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.749273    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-envoy\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.756733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.754596    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-certs\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.756733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.754892    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-data\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.756733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.754916    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-socket\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.756733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.755102    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-credential-socket\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.757456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.757430    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10996acb-8603-4451-804e-d808664017d5-istio-podinfo\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.760593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.760564    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10996acb-8603-4451-804e-d808664017d5-istiod-ca-cert\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.773234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.773201    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-istio-token\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.789700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.776188    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-envoy\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:45.801397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:45.801322    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gcz\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-kube-api-access-g4gcz\") pod \"e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") " pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:46.077143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:46.077054    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:46.371404 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:23:46.371370    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10996acb_8603_4451_804e_d808664017d5.slice/crio-0528fc81d253318575f6a7e0451f3f14c89900f46fde84e6c0b97eb9b9730de2 WatchSource:0}: Error finding container 0528fc81d253318575f6a7e0451f3f14c89900f46fde84e6c0b97eb9b9730de2: Status 404 returned error can't find the container with id 0528fc81d253318575f6a7e0451f3f14c89900f46fde84e6c0b97eb9b9730de2
Oct 30 21:23:46.371619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:46.371595    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"]
Oct 30 21:23:46.374111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:46.374071    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:46.374210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:46.374138    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:46.374210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:46.374167    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:23:47.328216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:47.328180    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" event={"ID":"10996acb-8603-4451-804e-d808664017d5","Type":"ContainerStarted","Data":"80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984"}
Oct 30 21:23:47.328635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:47.328244    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" event={"ID":"10996acb-8603-4451-804e-d808664017d5","Type":"ContainerStarted","Data":"0528fc81d253318575f6a7e0451f3f14c89900f46fde84e6c0b97eb9b9730de2"}
Oct 30 21:23:47.376663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:47.376595    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podStartSLOduration=2.376577895 podStartE2EDuration="2.376577895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:47.370245471 +0000 UTC m=+1335.719972892" watchObservedRunningTime="2026-10-30 21:23:47.376577895 +0000 UTC m=+1335.726305290"
Oct 30 21:23:48.077480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.077444    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:48.083845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.083815    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:48.251689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.251637    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s" podUID="fd1121df-e361-4e6b-8953-05c737617b9e" containerName="payload-processing" containerID="cri-o://263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7" gracePeriod=25
Oct 30 21:23:48.262214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.262174    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn" podUID="5f516526-0c1d-4adf-8500-75aa668e2f1e" containerName="payload-pre-processing" containerID="cri-o://05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95" gracePeriod=25
Oct 30 21:23:48.336660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.336577    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:48.337900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.337881    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:23:48.633581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.633546    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:48.673463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.673425    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1121df-e361-4e6b-8953-05c737617b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1121df-e361-4e6b-8953-05c737617b9e-plugins-config-volume\") pod \"fd1121df-e361-4e6b-8953-05c737617b9e\" (UID: \"fd1121df-e361-4e6b-8953-05c737617b9e\") "
Oct 30 21:23:48.673762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.673738    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1121df-e361-4e6b-8953-05c737617b9e-kube-api-access-wk6mn\" (UniqueName: \"kubernetes.io/projected/fd1121df-e361-4e6b-8953-05c737617b9e-kube-api-access-wk6mn\") pod \"fd1121df-e361-4e6b-8953-05c737617b9e\" (UID: \"fd1121df-e361-4e6b-8953-05c737617b9e\") "
Oct 30 21:23:48.673952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.673931    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1121df-e361-4e6b-8953-05c737617b9e-plugins-config-volume" pod "fd1121df-e361-4e6b-8953-05c737617b9e" (UID: "fd1121df-e361-4e6b-8953-05c737617b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:48.674501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.674475    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1121df-e361-4e6b-8953-05c737617b9e-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.676115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.676092    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1121df-e361-4e6b-8953-05c737617b9e-kube-api-access-wk6mn" pod "fd1121df-e361-4e6b-8953-05c737617b9e" (UID: "fd1121df-e361-4e6b-8953-05c737617b9e"). InnerVolumeSpecName "kube-api-access-wk6mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:48.681748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.681724    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:48.775247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.775208    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f516526-0c1d-4adf-8500-75aa668e2f1e-kube-api-access-df92x\" (UniqueName: \"kubernetes.io/projected/5f516526-0c1d-4adf-8500-75aa668e2f1e-kube-api-access-df92x\") pod \"5f516526-0c1d-4adf-8500-75aa668e2f1e\" (UID: \"5f516526-0c1d-4adf-8500-75aa668e2f1e\") "
Oct 30 21:23:48.775439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.775301    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f516526-0c1d-4adf-8500-75aa668e2f1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f516526-0c1d-4adf-8500-75aa668e2f1e-plugins-config-volume\") pod \"5f516526-0c1d-4adf-8500-75aa668e2f1e\" (UID: \"5f516526-0c1d-4adf-8500-75aa668e2f1e\") "
Oct 30 21:23:48.775763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.775735    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f516526-0c1d-4adf-8500-75aa668e2f1e-plugins-config-volume" pod "5f516526-0c1d-4adf-8500-75aa668e2f1e" (UID: "5f516526-0c1d-4adf-8500-75aa668e2f1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:48.776044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.776022    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6mn\" (UniqueName: \"kubernetes.io/projected/fd1121df-e361-4e6b-8953-05c737617b9e-kube-api-access-wk6mn\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.776174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.776047    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f516526-0c1d-4adf-8500-75aa668e2f1e-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.778110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.778084    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f516526-0c1d-4adf-8500-75aa668e2f1e-kube-api-access-df92x" pod "5f516526-0c1d-4adf-8500-75aa668e2f1e" (UID: "5f516526-0c1d-4adf-8500-75aa668e2f1e"). InnerVolumeSpecName "kube-api-access-df92x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:48.785221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.785193    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"]
Oct 30 21:23:48.785404 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.785378    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy" containerID="cri-o://9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef" gracePeriod=30
Oct 30 21:23:48.877447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:48.877366    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df92x\" (UniqueName: \"kubernetes.io/projected/5f516526-0c1d-4adf-8500-75aa668e2f1e-kube-api-access-df92x\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:49.346684 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.346641    4633 generic.go:356] "Generic (PLEG): container finished" podID="5f516526-0c1d-4adf-8500-75aa668e2f1e" containerID="05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95" exitCode=0
Oct 30 21:23:49.347163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.346730    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"
Oct 30 21:23:49.347163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.346744    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn" event={"ID":"5f516526-0c1d-4adf-8500-75aa668e2f1e","Type":"ContainerDied","Data":"05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95"}
Oct 30 21:23:49.347163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.346803    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn" event={"ID":"5f516526-0c1d-4adf-8500-75aa668e2f1e","Type":"ContainerDied","Data":"2a3b8b508a5e3bf8aa2e2bf6e66eaf4939784e3d03ca69f79e86f9b89b4d3921"}
Oct 30 21:23:49.347163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.346832    4633 scope.go:122] "RemoveContainer" containerID="05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95"
Oct 30 21:23:49.348791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.348766    4633 generic.go:356] "Generic (PLEG): container finished" podID="fd1121df-e361-4e6b-8953-05c737617b9e" containerID="263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7" exitCode=0
Oct 30 21:23:49.348895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.348809    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"
Oct 30 21:23:49.348895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.348854    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s" event={"ID":"fd1121df-e361-4e6b-8953-05c737617b9e","Type":"ContainerDied","Data":"263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7"}
Oct 30 21:23:49.348987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.348908    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s" event={"ID":"fd1121df-e361-4e6b-8953-05c737617b9e","Type":"ContainerDied","Data":"2a39ebfc75b71a292b161d0cb7d6b1375d01d8be16422c4edf5fbc34b2f6fa70"}
Oct 30 21:23:49.360324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.360298    4633 scope.go:122] "RemoveContainer" containerID="05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95"
Oct 30 21:23:49.360743 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:49.360713    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95\": container with ID starting with 05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95 not found: ID does not exist" containerID="05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95"
Oct 30 21:23:49.360842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.360748    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95"} err="failed to get container status \"05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95\": rpc error: code = NotFound desc = could not find container \"05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95\": container with ID starting with 05417b941b860d158b012514fc156070aff2e3c428e0e5dd880e5d67cc694a95 not found: ID does not exist"
Oct 30 21:23:49.360842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.360766    4633 scope.go:122] "RemoveContainer" containerID="263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7"
Oct 30 21:23:49.370995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.370976    4633 scope.go:122] "RemoveContainer" containerID="263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7"
Oct 30 21:23:49.371336 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:49.371315    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7\": container with ID starting with 263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7 not found: ID does not exist" containerID="263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7"
Oct 30 21:23:49.371414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.371348    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7"} err="failed to get container status \"263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7\": rpc error: code = NotFound desc = could not find container \"263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7\": container with ID starting with 263c5a06a2b5848109f20f78089ce6dd309f3a119baa2513106f3162b6a847e7 not found: ID does not exist"
Oct 30 21:23:49.379367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.379336    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"]
Oct 30 21:23:49.386064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.386039    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-69757d9dcf-ntv7s"]
Oct 30 21:23:49.399837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.399809    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"]
Oct 30 21:23:49.404607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.404571    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-66c746f4d7-2gqqn"]
Oct 30 21:23:49.792204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.792175    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f516526-0c1d-4adf-8500-75aa668e2f1e" path="/var/lib/kubelet/pods/5f516526-0c1d-4adf-8500-75aa668e2f1e/volumes"
Oct 30 21:23:49.792650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:49.792636    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1121df-e361-4e6b-8953-05c737617b9e" path="/var/lib/kubelet/pods/fd1121df-e361-4e6b-8953-05c737617b9e/volumes"
Oct 30 21:23:53.297792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.297756    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"]
Oct 30 21:23:53.298493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.298477    4633 cpu_manager.go:414] "containerMap: removing container" podUID="fd1121df-e361-4e6b-8953-05c737617b9e" containerName="payload-processing"
Oct 30 21:23:53.298556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.298495    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1121df-e361-4e6b-8953-05c737617b9e" containerName="payload-processing"
Oct 30 21:23:53.298556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.298510    4633 cpu_manager.go:414] "containerMap: removing container" podUID="5f516526-0c1d-4adf-8500-75aa668e2f1e" containerName="payload-pre-processing"
Oct 30 21:23:53.298556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.298517    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f516526-0c1d-4adf-8500-75aa668e2f1e" containerName="payload-pre-processing"
Oct 30 21:23:53.298660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.298613    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1121df-e361-4e6b-8953-05c737617b9e" containerName="payload-processing"
Oct 30 21:23:53.298660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.298622    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f516526-0c1d-4adf-8500-75aa668e2f1e" containerName="payload-pre-processing"
Oct 30 21:23:53.303869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.303850    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:53.310898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.310866    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0fc028\""
Oct 30 21:23:53.311617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.311590    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0fc028-dockercfg-sz4gh\""
Oct 30 21:23:53.327545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.324955    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"]
Oct 30 21:23:53.464695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.449743    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46kkq\" (UniqueName: \"kubernetes.io/projected/dde1ca2c-3c92-447d-881e-443fffc4d519-kube-api-access-46kkq\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb\" (UID: \"dde1ca2c-3c92-447d-881e-443fffc4d519\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:53.481699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.474802    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde1ca2c-3c92-447d-881e-443fffc4d519-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb\" (UID: \"dde1ca2c-3c92-447d-881e-443fffc4d519\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:53.578782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.577766    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde1ca2c-3c92-447d-881e-443fffc4d519-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb\" (UID: \"dde1ca2c-3c92-447d-881e-443fffc4d519\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:53.578782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.577927    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46kkq\" (UniqueName: \"kubernetes.io/projected/dde1ca2c-3c92-447d-881e-443fffc4d519-kube-api-access-46kkq\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb\" (UID: \"dde1ca2c-3c92-447d-881e-443fffc4d519\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:53.583780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.580715    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde1ca2c-3c92-447d-881e-443fffc4d519-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb\" (UID: \"dde1ca2c-3c92-447d-881e-443fffc4d519\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:53.595801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.593912    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46kkq\" (UniqueName: \"kubernetes.io/projected/dde1ca2c-3c92-447d-881e-443fffc4d519-kube-api-access-46kkq\") pod \"payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb\" (UID: \"dde1ca2c-3c92-447d-881e-443fffc4d519\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:53.675697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:53.659974    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:54.061135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.061084    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"]
Oct 30 21:23:54.066663 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:23:54.066625    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde1ca2c_3c92_447d_881e_443fffc4d519.slice/crio-48b4ae8c14e5954d8dbd4bf71c97bfdb49b0383f2ea526d54174928a74ebb8a2 WatchSource:0}: Error finding container 48b4ae8c14e5954d8dbd4bf71c97bfdb49b0383f2ea526d54174928a74ebb8a2: Status 404 returned error can't find the container with id 48b4ae8c14e5954d8dbd4bf71c97bfdb49b0383f2ea526d54174928a74ebb8a2
Oct 30 21:23:54.243982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.243938    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:23:54.362305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.362206    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-certs\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.362305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.362287    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-kube-api-access-nvwzl\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-kube-api-access-nvwzl\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.362870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.362403    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-data\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.362870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.362480    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-credential-socket\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.362870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.362632    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-token\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-token\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.362870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.362725    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istiod-ca-cert\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.362870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.362783    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-socket\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.362870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.362844    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-podinfo\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.363180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.363048    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-envoy\") pod \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\" (UID: \"70c1fb3c-70ac-41da-b1e3-86e307fc9369\") "
Oct 30 21:23:54.367794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.367158    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-certs" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.367794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.367453    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-data" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.367794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.367523    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-socket" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.367794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.367657    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istiod-ca-cert" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:54.367794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.367760    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-credential-socket" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.369098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.369072    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-kube-api-access-nvwzl" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "kube-api-access-nvwzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:54.369457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.369434    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-podinfo" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:54.369502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.369448    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-envoy" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.369837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.369809    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-token" pod "70c1fb3c-70ac-41da-b1e3-86e307fc9369" (UID: "70c1fb3c-70ac-41da-b1e3-86e307fc9369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470388    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvwzl\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-kube-api-access-nvwzl\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470428    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470442    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470456    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470475    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470490    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470504    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470515    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.470707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.470532    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c1fb3c-70ac-41da-b1e3-86e307fc9369-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.591169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.591134    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb" event={"ID":"dde1ca2c-3c92-447d-881e-443fffc4d519","Type":"ContainerStarted","Data":"0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf"}
Oct 30 21:23:54.591378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.591219    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb" event={"ID":"dde1ca2c-3c92-447d-881e-443fffc4d519","Type":"ContainerStarted","Data":"48b4ae8c14e5954d8dbd4bf71c97bfdb49b0383f2ea526d54174928a74ebb8a2"}
Oct 30 21:23:54.591459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.591376    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:23:54.592749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.592725    4633 generic.go:356] "Generic (PLEG): container finished" podID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerID="9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef" exitCode=0
Oct 30 21:23:54.592871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.592761    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" event={"ID":"70c1fb3c-70ac-41da-b1e3-86e307fc9369","Type":"ContainerDied","Data":"9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef"}
Oct 30 21:23:54.592871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.592792    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"
Oct 30 21:23:54.592871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.592821    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx" event={"ID":"70c1fb3c-70ac-41da-b1e3-86e307fc9369","Type":"ContainerDied","Data":"1ab25d33b2499f1a49a05a63a5c0e2f4a954057c028ddfe8df31fe005d279bcb"}
Oct 30 21:23:54.592871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.592852    4633 scope.go:122] "RemoveContainer" containerID="9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef"
Oct 30 21:23:54.647409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.647356    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb" podStartSLOduration=1.647339774 podStartE2EDuration="1.647339774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:54.624280188 +0000 UTC m=+1342.974007587" watchObservedRunningTime="2026-10-30 21:23:54.647339774 +0000 UTC m=+1342.997067170"
Oct 30 21:23:54.648351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.648327    4633 scope.go:122] "RemoveContainer" containerID="9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef"
Oct 30 21:23:54.648806 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:54.648774    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef\": container with ID starting with 9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef not found: ID does not exist" containerID="9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef"
Oct 30 21:23:54.648889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.648817    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef"} err="failed to get container status \"9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef\": rpc error: code = NotFound desc = could not find container \"9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef\": container with ID starting with 9afc9b1d0ac6a7bc9729557af73b5f2ff80b0db1380ca3bbcd0c8ea4844dd1ef not found: ID does not exist"
Oct 30 21:23:54.665606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.665572    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"]
Oct 30 21:23:54.672495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:54.672466    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-4jdgx"]
Oct 30 21:23:55.044695 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:23:55.044496    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:23:55.794087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:55.794057    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" path="/var/lib/kubelet/pods/70c1fb3c-70ac-41da-b1e3-86e307fc9369/volumes"
Oct 30 21:23:58.372949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.372912    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"]
Oct 30 21:23:58.374951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.374921    4633 cpu_manager.go:414] "containerMap: removing container" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy"
Oct 30 21:23:58.374951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.374952    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy"
Oct 30 21:23:58.377253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.377227    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c1fb3c-70ac-41da-b1e3-86e307fc9369" containerName="istio-proxy"
Oct 30 21:23:58.393083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.393051    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"]
Oct 30 21:23:58.393276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.393194    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.397905 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.397872    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-18b363-serving-cert\""
Oct 30 21:23:58.450528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.450491    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-maas-api-tls\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.450685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.450554    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdhck\" (UniqueName: \"kubernetes.io/projected/52aedae1-e213-4c88-bf1c-d21d4d563208-kube-api-access-rdhck\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.450685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.450646    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aedae1-e213-4c88-bf1c-d21d4d563208-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.450785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.450754    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-metrics-certs\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.554026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.553934    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-maas-api-tls\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.554289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.554263    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdhck\" (UniqueName: \"kubernetes.io/projected/52aedae1-e213-4c88-bf1c-d21d4d563208-kube-api-access-rdhck\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.556531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.556505    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aedae1-e213-4c88-bf1c-d21d4d563208-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.557724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.557700    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-metrics-certs\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.558338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.557537    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aedae1-e213-4c88-bf1c-d21d4d563208-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.569203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.569173    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-maas-api-tls\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.570160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.570128    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdhck\" (UniqueName: \"kubernetes.io/projected/52aedae1-e213-4c88-bf1c-d21d4d563208-kube-api-access-rdhck\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.572414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.571914    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-metrics-certs\") pod \"maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:58.715943 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:58.715904    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:59.046488 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:23:59.046315    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52aedae1_e213_4c88_bf1c_d21d4d563208.slice/crio-1930df429b5191ae93739af30c186e2b026ebced0c7642e5eaca83738c310468 WatchSource:0}: Error finding container 1930df429b5191ae93739af30c186e2b026ebced0c7642e5eaca83738c310468: Status 404 returned error can't find the container with id 1930df429b5191ae93739af30c186e2b026ebced0c7642e5eaca83738c310468
Oct 30 21:23:59.060859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:59.059433    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"]
Oct 30 21:23:59.660569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:59.660519    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5" event={"ID":"52aedae1-e213-4c88-bf1c-d21d4d563208","Type":"ContainerStarted","Data":"338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a"}
Oct 30 21:23:59.661059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:59.660586    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5" event={"ID":"52aedae1-e213-4c88-bf1c-d21d4d563208","Type":"ContainerStarted","Data":"1930df429b5191ae93739af30c186e2b026ebced0c7642e5eaca83738c310468"}
Oct 30 21:23:59.661059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:59.660653    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:23:59.684187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:23:59.684128    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5" podStartSLOduration=1.213367152 podStartE2EDuration="1.684112458s" totalImagesPullingTime="470.745306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:59.678660157 +0000 UTC m=+1348.028387746" watchObservedRunningTime="2026-10-30 21:23:59.684112458 +0000 UTC m=+1348.033839853"
Oct 30 21:24:02.754010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:02.753917    4633 generic.go:356] "Generic (PLEG): container finished" podID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerID="60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19" exitCode=0
Oct 30 21:24:02.754010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:02.753989    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" event={"ID":"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b","Type":"ContainerDied","Data":"60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19"}
Oct 30 21:24:02.754565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:02.754462    4633 scope.go:122] "RemoveContainer" containerID="60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19"
Oct 30 21:24:02.756004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:02.755973    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:02.756093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:02.756040    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:02.756093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:02.756072    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:03.763622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:03.763590    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" event={"ID":"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b","Type":"ContainerStarted","Data":"709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d"}
Oct 30 21:24:04.000162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:04.000129    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:24:04.001059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:04.001037    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:24:04.001177 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:04.001080    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 21:24:05.007696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:05.007244    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:24:05.007696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:05.007314    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 21:24:05.605218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:05.605187    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:24:05.747960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:05.747932    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:24:06.000142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:06.000113    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:24:06.000353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:06.000164    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 21:24:06.710866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:06.710837    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:24:07.000926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:07.000847    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:24:07.000926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:07.000904    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 21:24:08.004130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:08.004081    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:24:08.004803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:08.004767    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 21:24:09.008744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:09.006397    4633 patch_prober.go:29] interesting pod/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:24:09.008744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:09.006458    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 21:24:10.008053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:10.008017    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:24:10.009332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:10.009312    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:24:12.269778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:12.269745    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"]
Oct 30 21:24:12.270316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:12.270020    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy" containerID="cri-o://4efbb86417c450fe9a827199cef79c90a604a34ca660c1e4e843551e1ad1978f" gracePeriod=30
Oct 30 21:24:12.628601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:12.628519    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:24:16.982457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:16.982416    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"]
Oct 30 21:24:16.988916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:16.988882    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:16.994251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:16.994220    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"]
Oct 30 21:24:17.043173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.043141    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fde255-dockercfg-fjhvs\""
Oct 30 21:24:17.049741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.043527    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fde255\""
Oct 30 21:24:17.484509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.484468    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf23192-8e69-44f3-a81f-4037f9163b8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz\" (UID: \"9bf23192-8e69-44f3-a81f-4037f9163b8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:17.484946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.484913    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqt9n\" (UniqueName: \"kubernetes.io/projected/9bf23192-8e69-44f3-a81f-4037f9163b8b-kube-api-access-rqt9n\") pod \"payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz\" (UID: \"9bf23192-8e69-44f3-a81f-4037f9163b8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:17.507243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.505533    4633 generic.go:356] "Generic (PLEG): container finished" podID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerID="4efbb86417c450fe9a827199cef79c90a604a34ca660c1e4e843551e1ad1978f" exitCode=0
Oct 30 21:24:17.507243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.505923    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" event={"ID":"4f416feb-13be-4a2c-b2bc-9999484d1d56","Type":"ContainerDied","Data":"4efbb86417c450fe9a827199cef79c90a604a34ca660c1e4e843551e1ad1978f"}
Oct 30 21:24:17.507243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.506025    4633 scope.go:122] "RemoveContainer" containerID="534b1fa378bfe125e4d4b529ead2b1c62a51ce3ed36771f18b89393062f1dfb5"
Oct 30 21:24:17.586587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.586500    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf23192-8e69-44f3-a81f-4037f9163b8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz\" (UID: \"9bf23192-8e69-44f3-a81f-4037f9163b8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:17.586772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.586642    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt9n\" (UniqueName: \"kubernetes.io/projected/9bf23192-8e69-44f3-a81f-4037f9163b8b-kube-api-access-rqt9n\") pod \"payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz\" (UID: \"9bf23192-8e69-44f3-a81f-4037f9163b8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:17.587536 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.587509    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf23192-8e69-44f3-a81f-4037f9163b8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz\" (UID: \"9bf23192-8e69-44f3-a81f-4037f9163b8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:17.600019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.599308    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqt9n\" (UniqueName: \"kubernetes.io/projected/9bf23192-8e69-44f3-a81f-4037f9163b8b-kube-api-access-rqt9n\") pod \"payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz\" (UID: \"9bf23192-8e69-44f3-a81f-4037f9163b8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:17.611490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.611408    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:17.760852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.760824    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:24:17.789394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.788960    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-credential-socket\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.789394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789018    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f416feb-13be-4a2c-b2bc-9999484d1d56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f416feb-13be-4a2c-b2bc-9999484d1d56-istiod-ca-cert\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.789394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789066    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-envoy\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.789394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789218    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-kube-api-access-7vz8n\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-kube-api-access-7vz8n\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.789394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789260    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-socket\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.789394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789321    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-token\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-token\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.789394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789370    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-data\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.789394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789399    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-podinfo\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.790031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789489    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-certs\") pod \"4f416feb-13be-4a2c-b2bc-9999484d1d56\" (UID: \"4f416feb-13be-4a2c-b2bc-9999484d1d56\") "
Oct 30 21:24:17.790031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789524    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f416feb-13be-4a2c-b2bc-9999484d1d56-istiod-ca-cert" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:17.790031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789765    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-certs" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.790031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.789947    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-data" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.790234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.790164    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-socket" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.791205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.790434    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-credential-socket" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.791205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.791063    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.791205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.791082    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.791205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.791096    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f416feb-13be-4a2c-b2bc-9999484d1d56-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.791205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.791108    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.791205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.791122    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.792553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.792294    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-kube-api-access-7vz8n" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "kube-api-access-7vz8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:17.792691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.792579    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-token" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:17.793815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.793793    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-podinfo" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:24:17.794729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.794705    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-envoy" pod "4f416feb-13be-4a2c-b2bc-9999484d1d56" (UID: "4f416feb-13be-4a2c-b2bc-9999484d1d56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:17.892872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.892790    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.892872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.892821    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vz8n\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-kube-api-access-7vz8n\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.892872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.892832    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.892872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.892840    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f416feb-13be-4a2c-b2bc-9999484d1d56-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:17.961693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:17.961647    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"]
Oct 30 21:24:17.961814 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:24:17.961791    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf23192_8e69_44f3_a81f_4037f9163b8b.slice/crio-856cb94c71ee5ad24e3978999ecbb2718aa3a1a74d638948348aba77809f5f3c WatchSource:0}: Error finding container 856cb94c71ee5ad24e3978999ecbb2718aa3a1a74d638948348aba77809f5f3c: Status 404 returned error can't find the container with id 856cb94c71ee5ad24e3978999ecbb2718aa3a1a74d638948348aba77809f5f3c
Oct 30 21:24:18.539521 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.539480    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb" event={"ID":"4f416feb-13be-4a2c-b2bc-9999484d1d56","Type":"ContainerDied","Data":"109ed81ab0fcf4431a9403e8fc10d02bf4dcd05505ab142575e2475447c99d40"}
Oct 30 21:24:18.540086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.539498    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"
Oct 30 21:24:18.540086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.539555    4633 scope.go:122] "RemoveContainer" containerID="4efbb86417c450fe9a827199cef79c90a604a34ca660c1e4e843551e1ad1978f"
Oct 30 21:24:18.542819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.542651    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz" event={"ID":"9bf23192-8e69-44f3-a81f-4037f9163b8b","Type":"ContainerStarted","Data":"dd3f83a1e938e41fc765b873abe480585159b70d163946d90c19ab03e46fa69d"}
Oct 30 21:24:18.542963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.542850    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz" event={"ID":"9bf23192-8e69-44f3-a81f-4037f9163b8b","Type":"ContainerStarted","Data":"856cb94c71ee5ad24e3978999ecbb2718aa3a1a74d638948348aba77809f5f3c"}
Oct 30 21:24:18.543033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.543008    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:18.565191 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.565146    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz" podStartSLOduration=2.565132935 podStartE2EDuration="2.565132935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:18.564329404 +0000 UTC m=+1366.914056805" watchObservedRunningTime="2026-10-30 21:24:18.565132935 +0000 UTC m=+1366.914860331"
Oct 30 21:24:18.580462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.580432    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"]
Oct 30 21:24:18.586204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:18.586179    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f6ab-openshift-default-6c59845c65-zwknb"]
Oct 30 21:24:19.792857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:19.792814    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" path="/var/lib/kubelet/pods/4f416feb-13be-4a2c-b2bc-9999484d1d56/volumes"
Oct 30 21:24:20.145415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.145324    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"]
Oct 30 21:24:20.146205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.146177    4633 cpu_manager.go:414] "containerMap: removing container" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy"
Oct 30 21:24:20.146205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.146205    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy"
Oct 30 21:24:20.146533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.146223    4633 cpu_manager.go:414] "containerMap: removing container" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy"
Oct 30 21:24:20.146533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.146231    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy"
Oct 30 21:24:20.146533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.146432    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy"
Oct 30 21:24:20.146533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.146449    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f416feb-13be-4a2c-b2bc-9999484d1d56" containerName="istio-proxy"
Oct 30 21:24:20.150738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.150711    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"]
Oct 30 21:24:20.150878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.150793    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:20.155180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.155157    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7db332\""
Oct 30 21:24:20.155305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.155221    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7db332-dockercfg-jkktx\""
Oct 30 21:24:20.172768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.172740    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"]
Oct 30 21:24:20.177256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.177231    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:20.189987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.189960    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"]
Oct 30 21:24:20.474834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.474793    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnb7\" (UniqueName: \"kubernetes.io/projected/495f00f7-d937-44d7-b15d-0cfa16359a69-kube-api-access-xnnb7\") pod \"payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2\" (UID: \"495f00f7-d937-44d7-b15d-0cfa16359a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:20.475295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.475271    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edf50f5-bdf9-4268-9b4d-228f49b1910d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm\" (UID: \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:20.475546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.475527    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfz6v\" (UniqueName: \"kubernetes.io/projected/1edf50f5-bdf9-4268-9b4d-228f49b1910d-kube-api-access-kfz6v\") pod \"payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm\" (UID: \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:20.475778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.475743    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495f00f7-d937-44d7-b15d-0cfa16359a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2\" (UID: \"495f00f7-d937-44d7-b15d-0cfa16359a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:20.580047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.579947    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnb7\" (UniqueName: \"kubernetes.io/projected/495f00f7-d937-44d7-b15d-0cfa16359a69-kube-api-access-xnnb7\") pod \"payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2\" (UID: \"495f00f7-d937-44d7-b15d-0cfa16359a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:20.580448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.580421    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edf50f5-bdf9-4268-9b4d-228f49b1910d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm\" (UID: \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:20.580712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.580690    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfz6v\" (UniqueName: \"kubernetes.io/projected/1edf50f5-bdf9-4268-9b4d-228f49b1910d-kube-api-access-kfz6v\") pod \"payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm\" (UID: \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:20.603017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.602983    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495f00f7-d937-44d7-b15d-0cfa16359a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2\" (UID: \"495f00f7-d937-44d7-b15d-0cfa16359a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:20.604045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.582754    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edf50f5-bdf9-4268-9b4d-228f49b1910d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm\" (UID: \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:20.606189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.604105    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495f00f7-d937-44d7-b15d-0cfa16359a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2\" (UID: \"495f00f7-d937-44d7-b15d-0cfa16359a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:20.607399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.607371    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnb7\" (UniqueName: \"kubernetes.io/projected/495f00f7-d937-44d7-b15d-0cfa16359a69-kube-api-access-xnnb7\") pod \"payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2\" (UID: \"495f00f7-d937-44d7-b15d-0cfa16359a69\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:20.611475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.611450    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfz6v\" (UniqueName: \"kubernetes.io/projected/1edf50f5-bdf9-4268-9b4d-228f49b1910d-kube-api-access-kfz6v\") pod \"payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm\" (UID: \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:20.771815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.771713    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:20.796199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:20.796166    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:21.066009 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:24:21.065979    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495f00f7_d937_44d7_b15d_0cfa16359a69.slice/crio-c33938384770a9e1f3f59e560f752274347e61b0773a49458c56a389987a6e1c WatchSource:0}: Error finding container c33938384770a9e1f3f59e560f752274347e61b0773a49458c56a389987a6e1c: Status 404 returned error can't find the container with id c33938384770a9e1f3f59e560f752274347e61b0773a49458c56a389987a6e1c
Oct 30 21:24:21.066617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.066590    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"]
Oct 30 21:24:21.114151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.114129    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"]
Oct 30 21:24:21.115539 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:24:21.115503    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edf50f5_bdf9_4268_9b4d_228f49b1910d.slice/crio-e819b75d337745760c922dac876cc394227d9e931c304ae8cc6e724ec90a0b56 WatchSource:0}: Error finding container e819b75d337745760c922dac876cc394227d9e931c304ae8cc6e724ec90a0b56: Status 404 returned error can't find the container with id e819b75d337745760c922dac876cc394227d9e931c304ae8cc6e724ec90a0b56
Oct 30 21:24:21.625356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.625314    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2" event={"ID":"495f00f7-d937-44d7-b15d-0cfa16359a69","Type":"ContainerStarted","Data":"a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a"}
Oct 30 21:24:21.625918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.625886    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:21.626074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.625947    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2" event={"ID":"495f00f7-d937-44d7-b15d-0cfa16359a69","Type":"ContainerStarted","Data":"c33938384770a9e1f3f59e560f752274347e61b0773a49458c56a389987a6e1c"}
Oct 30 21:24:21.627804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.627774    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm" event={"ID":"1edf50f5-bdf9-4268-9b4d-228f49b1910d","Type":"ContainerStarted","Data":"47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762"}
Oct 30 21:24:21.627920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.627827    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm" event={"ID":"1edf50f5-bdf9-4268-9b4d-228f49b1910d","Type":"ContainerStarted","Data":"e819b75d337745760c922dac876cc394227d9e931c304ae8cc6e724ec90a0b56"}
Oct 30 21:24:21.627920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.627875    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:21.648041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.647991    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2" podStartSLOduration=1.647977569 podStartE2EDuration="1.647977569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:21.64417107 +0000 UTC m=+1369.993898469" watchObservedRunningTime="2026-10-30 21:24:21.647977569 +0000 UTC m=+1369.997704964"
Oct 30 21:24:21.665604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:21.665545    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm" podStartSLOduration=1.665529393 podStartE2EDuration="1.665529393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:21.660955505 +0000 UTC m=+1370.010682902" watchObservedRunningTime="2026-10-30 21:24:21.665529393 +0000 UTC m=+1370.015256789"
Oct 30 21:24:24.875506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:24.875460    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"]
Oct 30 21:24:24.881694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:24.881650    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:24.891728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:24.890965    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-fde255-serving-cert\""
Oct 30 21:24:24.891728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:24.891534    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"]
Oct 30 21:24:24.993887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:24.993835    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:24.994064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:24.993920    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzx6w\" (UniqueName: \"kubernetes.io/projected/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-kube-api-access-pzx6w\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:24.994133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:24.994103    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-metrics-certs\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:24.994349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:24.994327    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.021489 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:24:25.021445    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:24:25.108039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.102591    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.109223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.106651    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.109223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.108351    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzx6w\" (UniqueName: \"kubernetes.io/projected/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-kube-api-access-pzx6w\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.109223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.108441    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-metrics-certs\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.109223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.108577    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.109223 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:24:25.108927    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-serving-cert: secret "maas-api-e2e-ipp-b-fde255-serving-cert" not found
Oct 30 21:24:25.109223 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:24:25.108997    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls podName:0ca9dc71-d72d-456e-9c02-624e03cdc4ff nodeName:}" failed. No retries permitted until 2026-10-30 21:24:25.608973681 +0000 UTC m=+1373.958701058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls") pod "maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp" (UID: "0ca9dc71-d72d-456e-9c02-624e03cdc4ff") : secret "maas-api-e2e-ipp-b-fde255-serving-cert" not found
Oct 30 21:24:25.148588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.148464    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-metrics-certs\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.153273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.153240    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzx6w\" (UniqueName: \"kubernetes.io/projected/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-kube-api-access-pzx6w\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.629274 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.629165    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.632061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.632030    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:25.664607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.664519    4633 generic.go:356] "Generic (PLEG): container finished" podID="10996acb-8603-4451-804e-d808664017d5" containerID="80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984" exitCode=0
Oct 30 21:24:25.664806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.664609    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" event={"ID":"10996acb-8603-4451-804e-d808664017d5","Type":"ContainerDied","Data":"80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984"}
Oct 30 21:24:25.665155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.665140    4633 scope.go:122] "RemoveContainer" containerID="80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984"
Oct 30 21:24:25.666963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.666936    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:25.667038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.667005    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:25.667074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.667035    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:25.809484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:25.808177    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:26.123625 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:24:26.123591    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca9dc71_d72d_456e_9c02_624e03cdc4ff.slice/crio-22fa1f3ca941723d4c9e74f7cd3836c5dd2509993f4144e229f8308cf9d021b9 WatchSource:0}: Error finding container 22fa1f3ca941723d4c9e74f7cd3836c5dd2509993f4144e229f8308cf9d021b9: Status 404 returned error can't find the container with id 22fa1f3ca941723d4c9e74f7cd3836c5dd2509993f4144e229f8308cf9d021b9
Oct 30 21:24:26.128858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:26.127338    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"]
Oct 30 21:24:26.709057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:26.705522    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" event={"ID":"10996acb-8603-4451-804e-d808664017d5","Type":"ContainerStarted","Data":"83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2"}
Oct 30 21:24:26.774659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:26.774619    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp" event={"ID":"0ca9dc71-d72d-456e-9c02-624e03cdc4ff","Type":"ContainerStarted","Data":"22fa1f3ca941723d4c9e74f7cd3836c5dd2509993f4144e229f8308cf9d021b9"}
Oct 30 21:24:27.083692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:27.078016    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:24:27.083692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:27.079303    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:24:27.083692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:27.079357    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 21:24:27.793294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:27.793260    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:27.793817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:27.793301    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp" event={"ID":"0ca9dc71-d72d-456e-9c02-624e03cdc4ff","Type":"ContainerStarted","Data":"b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09"}
Oct 30 21:24:27.803232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:27.803182    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp" podStartSLOduration=3.347186589 podStartE2EDuration="3.803168259s" totalImagesPullingTime="455.98167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:27.80276905 +0000 UTC m=+1376.152496448" watchObservedRunningTime="2026-10-30 21:24:27.803168259 +0000 UTC m=+1376.152895654"
Oct 30 21:24:28.077803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:28.077711    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:24:28.077803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:28.077786    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 21:24:28.969483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:28.965928    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"]
Oct 30 21:24:28.969483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:28.966364    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp" podUID="3a933c8c-6095-426d-aed1-0bdfcc92aa15" containerName="maas-api" containerID="cri-o://c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87" gracePeriod=30
Oct 30 21:24:29.078733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.077574    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:24:29.078733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.077642    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 21:24:29.559481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.558247    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:29.568004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.567971    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:24:29.624058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.610540    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-metrics-certs\") pod \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") "
Oct 30 21:24:29.624058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.610598    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a933c8c-6095-426d-aed1-0bdfcc92aa15-kube-api-access-7625v\" (UniqueName: \"kubernetes.io/projected/3a933c8c-6095-426d-aed1-0bdfcc92aa15-kube-api-access-7625v\") pod \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") "
Oct 30 21:24:29.624058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.610777    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a933c8c-6095-426d-aed1-0bdfcc92aa15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a933c8c-6095-426d-aed1-0bdfcc92aa15-openshift-service-ca\") pod \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") "
Oct 30 21:24:29.624058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.612080    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a933c8c-6095-426d-aed1-0bdfcc92aa15-openshift-service-ca" pod "3a933c8c-6095-426d-aed1-0bdfcc92aa15" (UID: "3a933c8c-6095-426d-aed1-0bdfcc92aa15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:29.624058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.612364    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-maas-api-tls\") pod \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\" (UID: \"3a933c8c-6095-426d-aed1-0bdfcc92aa15\") "
Oct 30 21:24:29.624058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.613490    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a933c8c-6095-426d-aed1-0bdfcc92aa15-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:29.650857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.649819    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-metrics-certs" pod "3a933c8c-6095-426d-aed1-0bdfcc92aa15" (UID: "3a933c8c-6095-426d-aed1-0bdfcc92aa15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:29.652862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.652831    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a933c8c-6095-426d-aed1-0bdfcc92aa15-kube-api-access-7625v" pod "3a933c8c-6095-426d-aed1-0bdfcc92aa15" (UID: "3a933c8c-6095-426d-aed1-0bdfcc92aa15"). InnerVolumeSpecName "kube-api-access-7625v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:29.653214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.653195    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-maas-api-tls" pod "3a933c8c-6095-426d-aed1-0bdfcc92aa15" (UID: "3a933c8c-6095-426d-aed1-0bdfcc92aa15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:29.739080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.739049    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:29.739080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.739079    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a933c8c-6095-426d-aed1-0bdfcc92aa15-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:29.739298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.739092    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7625v\" (UniqueName: \"kubernetes.io/projected/3a933c8c-6095-426d-aed1-0bdfcc92aa15-kube-api-access-7625v\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:29.959733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.959002    4633 generic.go:356] "Generic (PLEG): container finished" podID="3a933c8c-6095-426d-aed1-0bdfcc92aa15" containerID="c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87" exitCode=0
Oct 30 21:24:29.961096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.961067    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"
Oct 30 21:24:29.962409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.962382    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp" event={"ID":"3a933c8c-6095-426d-aed1-0bdfcc92aa15","Type":"ContainerDied","Data":"c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87"}
Oct 30 21:24:29.963453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.963429    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp" event={"ID":"3a933c8c-6095-426d-aed1-0bdfcc92aa15","Type":"ContainerDied","Data":"882137a54fa9d41921f37497dba75c68051910ea9f109df85bc72845f42aff94"}
Oct 30 21:24:29.963612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:29.963598    4633 scope.go:122] "RemoveContainer" containerID="c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87"
Oct 30 21:24:30.030162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:30.030135    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"]
Oct 30 21:24:30.030985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:30.030965    4633 scope.go:122] "RemoveContainer" containerID="c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87"
Oct 30 21:24:30.031648 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:24:30.031624    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87\": container with ID starting with c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87 not found: ID does not exist" containerID="c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87"
Oct 30 21:24:30.036785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:30.036750    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87"} err="failed to get container status \"c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87\": rpc error: code = NotFound desc = could not find container \"c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87\": container with ID starting with c06e455605ec337d4262e2fa7b7beb04197dc6fbd07d16019858b1a663564e87 not found: ID does not exist"
Oct 30 21:24:30.052386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:30.052302    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-83921a-78df9885b4-5fltp"]
Oct 30 21:24:30.084289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:30.083735    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:24:30.084585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:30.084551    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 21:24:31.082031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:31.081995    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:24:31.082619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:31.082583    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 21:24:31.792952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:31.792920    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a933c8c-6095-426d-aed1-0bdfcc92aa15" path="/var/lib/kubelet/pods/3a933c8c-6095-426d-aed1-0bdfcc92aa15/volumes"
Oct 30 21:24:32.088756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:32.085074    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Oct 30 21:24:32.088756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:32.085142    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Oct 30 21:24:32.641843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:32.641811    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:24:32.642003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:32.641903    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:24:33.082592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:33.082562    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:24:33.083071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:33.083051    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:24:33.083944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:33.083924    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:24:33.132816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:33.132786    4633 scope.go:122] "RemoveContainer" containerID="8e76caa43c7ae0ec4db876575371ef903eb02e024431b92879e9c29bbcf601bd"
Oct 30 21:24:34.967196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:34.967169    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:24:38.522514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:38.522481    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:24:40.165233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:40.165173    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"]
Oct 30 21:24:45.166162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.166070    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz" podUID="9bf23192-8e69-44f3-a81f-4037f9163b8b" containerName="payload-processing" containerID="cri-o://dd3f83a1e938e41fc765b873abe480585159b70d163946d90c19ab03e46fa69d" gracePeriod=25
Oct 30 21:24:45.415074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.415038    4633 generic.go:356] "Generic (PLEG): container finished" podID="9bf23192-8e69-44f3-a81f-4037f9163b8b" containerID="dd3f83a1e938e41fc765b873abe480585159b70d163946d90c19ab03e46fa69d" exitCode=0
Oct 30 21:24:45.415245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.415115    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz" event={"ID":"9bf23192-8e69-44f3-a81f-4037f9163b8b","Type":"ContainerDied","Data":"dd3f83a1e938e41fc765b873abe480585159b70d163946d90c19ab03e46fa69d"}
Oct 30 21:24:45.508183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.508152    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:45.551400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.551356    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf23192-8e69-44f3-a81f-4037f9163b8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf23192-8e69-44f3-a81f-4037f9163b8b-plugins-config-volume\") pod \"9bf23192-8e69-44f3-a81f-4037f9163b8b\" (UID: \"9bf23192-8e69-44f3-a81f-4037f9163b8b\") "
Oct 30 21:24:45.551590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.551563    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf23192-8e69-44f3-a81f-4037f9163b8b-kube-api-access-rqt9n\" (UniqueName: \"kubernetes.io/projected/9bf23192-8e69-44f3-a81f-4037f9163b8b-kube-api-access-rqt9n\") pod \"9bf23192-8e69-44f3-a81f-4037f9163b8b\" (UID: \"9bf23192-8e69-44f3-a81f-4037f9163b8b\") "
Oct 30 21:24:45.551915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.551860    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf23192-8e69-44f3-a81f-4037f9163b8b-plugins-config-volume" pod "9bf23192-8e69-44f3-a81f-4037f9163b8b" (UID: "9bf23192-8e69-44f3-a81f-4037f9163b8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:45.552906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.552880    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bf23192-8e69-44f3-a81f-4037f9163b8b-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:45.554089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.554061    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf23192-8e69-44f3-a81f-4037f9163b8b-kube-api-access-rqt9n" pod "9bf23192-8e69-44f3-a81f-4037f9163b8b" (UID: "9bf23192-8e69-44f3-a81f-4037f9163b8b"). InnerVolumeSpecName "kube-api-access-rqt9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:45.655113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:45.655070    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqt9n\" (UniqueName: \"kubernetes.io/projected/9bf23192-8e69-44f3-a81f-4037f9163b8b-kube-api-access-rqt9n\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.428858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.428629    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz" event={"ID":"9bf23192-8e69-44f3-a81f-4037f9163b8b","Type":"ContainerDied","Data":"856cb94c71ee5ad24e3978999ecbb2718aa3a1a74d638948348aba77809f5f3c"}
Oct 30 21:24:46.428858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.428726    4633 scope.go:122] "RemoveContainer" containerID="dd3f83a1e938e41fc765b873abe480585159b70d163946d90c19ab03e46fa69d"
Oct 30 21:24:46.429844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.429822    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"
Oct 30 21:24:46.459803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.453712    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"]
Oct 30 21:24:46.459803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.454879    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3a933c8c-6095-426d-aed1-0bdfcc92aa15" containerName="maas-api"
Oct 30 21:24:46.459803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.454902    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a933c8c-6095-426d-aed1-0bdfcc92aa15" containerName="maas-api"
Oct 30 21:24:46.459803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.454919    4633 cpu_manager.go:414] "containerMap: removing container" podUID="9bf23192-8e69-44f3-a81f-4037f9163b8b" containerName="payload-processing"
Oct 30 21:24:46.459803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.454927    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf23192-8e69-44f3-a81f-4037f9163b8b" containerName="payload-processing"
Oct 30 21:24:46.459803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.455140    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a933c8c-6095-426d-aed1-0bdfcc92aa15" containerName="maas-api"
Oct 30 21:24:46.459803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.455154    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf23192-8e69-44f3-a81f-4037f9163b8b" containerName="payload-processing"
Oct 30 21:24:46.473400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.473366    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"]
Oct 30 21:24:46.473600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.473497    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.487011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.486974    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"]
Oct 30 21:24:46.493652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.493621    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-6758c95d7c-gp5xz"]
Oct 30 21:24:46.590895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.590859    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.591092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.590904    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f3a996-275d-4731-81b7-d8e8c240f429-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.591092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.590974    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzsrv\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-kube-api-access-jzsrv\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.591092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.591003    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.591238 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.591095    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.591238 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.591157    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.591321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.591240    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.591321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.591299    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.591401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.591379    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.692819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.692708    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.692819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.692773    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.692819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.692814    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f3a996-275d-4731-81b7-d8e8c240f429-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.692953    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzsrv\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-kube-api-access-jzsrv\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693009    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693110    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693148    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693259    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693367    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693416    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693561    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693694    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f3a996-275d-4731-81b7-d8e8c240f429-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693706    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.693821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.693765    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.695860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.695836    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.696071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.696050    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.701657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.701632    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.701801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.701743    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzsrv\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-kube-api-access-jzsrv\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:46.807747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:46.807708    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:47.014456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.014382    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"]
Oct 30 21:24:47.019983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.019961    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:47.024808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.024752    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fde255\""
Oct 30 21:24:47.028319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.028291    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fde255-dockercfg-jtlpv\""
Oct 30 21:24:47.031248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.031225    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"]
Oct 30 21:24:47.103231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.101643    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/5c5ff786-1734-4d96-b69a-e0057ea851c7-kube-api-access-9tf4f\") pod \"payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9\" (UID: \"5c5ff786-1734-4d96-b69a-e0057ea851c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:47.103231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.101920    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5ff786-1734-4d96-b69a-e0057ea851c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9\" (UID: \"5c5ff786-1734-4d96-b69a-e0057ea851c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:47.132695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.132644    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"]
Oct 30 21:24:47.134576 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:24:47.134541    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f3a996_275d_4731_81b7_d8e8c240f429.slice/crio-ff9c8c730f1a4c92196260e93ec0d5af943b90e342f86fcea0af1aafe97a468d WatchSource:0}: Error finding container ff9c8c730f1a4c92196260e93ec0d5af943b90e342f86fcea0af1aafe97a468d: Status 404 returned error can't find the container with id ff9c8c730f1a4c92196260e93ec0d5af943b90e342f86fcea0af1aafe97a468d
Oct 30 21:24:47.137909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.137844    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:47.138043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.137948    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:47.138043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.137994    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:24:47.203250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.203214    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/5c5ff786-1734-4d96-b69a-e0057ea851c7-kube-api-access-9tf4f\") pod \"payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9\" (UID: \"5c5ff786-1734-4d96-b69a-e0057ea851c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:47.203440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.203424    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5ff786-1734-4d96-b69a-e0057ea851c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9\" (UID: \"5c5ff786-1734-4d96-b69a-e0057ea851c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:47.204230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.204208    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5ff786-1734-4d96-b69a-e0057ea851c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9\" (UID: \"5c5ff786-1734-4d96-b69a-e0057ea851c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:47.213567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.213531    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/5c5ff786-1734-4d96-b69a-e0057ea851c7-kube-api-access-9tf4f\") pod \"payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9\" (UID: \"5c5ff786-1734-4d96-b69a-e0057ea851c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:47.337975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.337898    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:47.444804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.443527    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" event={"ID":"a6f3a996-275d-4731-81b7-d8e8c240f429","Type":"ContainerStarted","Data":"d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f"}
Oct 30 21:24:47.444804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.443590    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" event={"ID":"a6f3a996-275d-4731-81b7-d8e8c240f429","Type":"ContainerStarted","Data":"ff9c8c730f1a4c92196260e93ec0d5af943b90e342f86fcea0af1aafe97a468d"}
Oct 30 21:24:47.749697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.749549    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podStartSLOduration=1.565650249 podStartE2EDuration="1.565650249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:47.564366431 +0000 UTC m=+1395.914093827" watchObservedRunningTime="2026-10-30 21:24:47.565650249 +0000 UTC m=+1395.915377645"
Oct 30 21:24:47.815383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.814853    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf23192-8e69-44f3-a81f-4037f9163b8b" path="/var/lib/kubelet/pods/9bf23192-8e69-44f3-a81f-4037f9163b8b/volumes"
Oct 30 21:24:47.815383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.815325    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:47.835219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:47.835181    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"]
Oct 30 21:24:47.917258 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:24:47.916490    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5ff786_1734_4d96_b69a_e0057ea851c7.slice/crio-5a8f39b81961eb7062d55de0a1b5675d4e2278675d8e41761d5767182b54a246 WatchSource:0}: Error finding container 5a8f39b81961eb7062d55de0a1b5675d4e2278675d8e41761d5767182b54a246: Status 404 returned error can't find the container with id 5a8f39b81961eb7062d55de0a1b5675d4e2278675d8e41761d5767182b54a246
Oct 30 21:24:48.460056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:48.460016    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9" event={"ID":"5c5ff786-1734-4d96-b69a-e0057ea851c7","Type":"ContainerStarted","Data":"e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0"}
Oct 30 21:24:48.460542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:48.460081    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9" event={"ID":"5c5ff786-1734-4d96-b69a-e0057ea851c7","Type":"ContainerStarted","Data":"5a8f39b81961eb7062d55de0a1b5675d4e2278675d8e41761d5767182b54a246"}
Oct 30 21:24:48.460542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:48.460367    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:24:48.481315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:48.481262    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9" podStartSLOduration=2.4812480089999998 podStartE2EDuration="2.481248009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:48.477837656 +0000 UTC m=+1396.827565053" watchObservedRunningTime="2026-10-30 21:24:48.481248009 +0000 UTC m=+1396.830975469"
Oct 30 21:24:48.809803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:48.809724    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:24:48.809803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:48.809773    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 21:24:49.809026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:49.808990    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:24:49.809522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:49.809055    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 21:24:50.728697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:50.714212    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"]
Oct 30 21:24:50.728697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:50.714464    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5" podUID="52aedae1-e213-4c88-bf1c-d21d4d563208" containerName="maas-api" containerID="cri-o://338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a" gracePeriod=30
Oct 30 21:24:50.751500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:50.751453    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5" podUID="52aedae1-e213-4c88-bf1c-d21d4d563208" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": dial tcp 10.133.0.161:8443: connect: connection refused"
Oct 30 21:24:50.809003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:50.808971    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:24:50.809160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:50.809030    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 21:24:51.455480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.455444    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:24:51.498989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.498149    4633 generic.go:356] "Generic (PLEG): container finished" podID="52aedae1-e213-4c88-bf1c-d21d4d563208" containerID="338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a" exitCode=0
Oct 30 21:24:51.498989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.498207    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5" event={"ID":"52aedae1-e213-4c88-bf1c-d21d4d563208","Type":"ContainerDied","Data":"338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a"}
Oct 30 21:24:51.498989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.498272    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5" event={"ID":"52aedae1-e213-4c88-bf1c-d21d4d563208","Type":"ContainerDied","Data":"1930df429b5191ae93739af30c186e2b026ebced0c7642e5eaca83738c310468"}
Oct 30 21:24:51.498989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.498295    4633 scope.go:122] "RemoveContainer" containerID="338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a"
Oct 30 21:24:51.498989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.498407    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"
Oct 30 21:24:51.510747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.510726    4633 scope.go:122] "RemoveContainer" containerID="338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a"
Oct 30 21:24:51.511204 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:24:51.511178    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a\": container with ID starting with 338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a not found: ID does not exist" containerID="338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a"
Oct 30 21:24:51.511315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.511217    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a"} err="failed to get container status \"338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a\": rpc error: code = NotFound desc = could not find container \"338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a\": container with ID starting with 338fbebaf2a3bfa0fed3cc6097a2e2fb9a06917f9064bbcf5c0aa7661f3a484a not found: ID does not exist"
Oct 30 21:24:51.533712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.512763    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52aedae1-e213-4c88-bf1c-d21d4d563208-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aedae1-e213-4c88-bf1c-d21d4d563208-openshift-service-ca\") pod \"52aedae1-e213-4c88-bf1c-d21d4d563208\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") "
Oct 30 21:24:51.533712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.512806    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-maas-api-tls\") pod \"52aedae1-e213-4c88-bf1c-d21d4d563208\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") "
Oct 30 21:24:51.533712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.512959    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-metrics-certs\") pod \"52aedae1-e213-4c88-bf1c-d21d4d563208\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") "
Oct 30 21:24:51.533712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.513011    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52aedae1-e213-4c88-bf1c-d21d4d563208-kube-api-access-rdhck\" (UniqueName: \"kubernetes.io/projected/52aedae1-e213-4c88-bf1c-d21d4d563208-kube-api-access-rdhck\") pod \"52aedae1-e213-4c88-bf1c-d21d4d563208\" (UID: \"52aedae1-e213-4c88-bf1c-d21d4d563208\") "
Oct 30 21:24:51.540744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.539835    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52aedae1-e213-4c88-bf1c-d21d4d563208-openshift-service-ca" pod "52aedae1-e213-4c88-bf1c-d21d4d563208" (UID: "52aedae1-e213-4c88-bf1c-d21d4d563208"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:51.547058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.547019    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-metrics-certs" pod "52aedae1-e213-4c88-bf1c-d21d4d563208" (UID: "52aedae1-e213-4c88-bf1c-d21d4d563208"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:51.548299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.548269    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52aedae1-e213-4c88-bf1c-d21d4d563208-kube-api-access-rdhck" pod "52aedae1-e213-4c88-bf1c-d21d4d563208" (UID: "52aedae1-e213-4c88-bf1c-d21d4d563208"). InnerVolumeSpecName "kube-api-access-rdhck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:51.548711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.548692    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-maas-api-tls" pod "52aedae1-e213-4c88-bf1c-d21d4d563208" (UID: "52aedae1-e213-4c88-bf1c-d21d4d563208"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:51.642699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.639189    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:51.642699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.639230    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdhck\" (UniqueName: \"kubernetes.io/projected/52aedae1-e213-4c88-bf1c-d21d4d563208-kube-api-access-rdhck\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:51.642699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.639247    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52aedae1-e213-4c88-bf1c-d21d4d563208-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:51.642699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.639262    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52aedae1-e213-4c88-bf1c-d21d4d563208-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:51.643785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.643652    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"]
Oct 30 21:24:51.644725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.644700    4633 cpu_manager.go:414] "containerMap: removing container" podUID="52aedae1-e213-4c88-bf1c-d21d4d563208" containerName="maas-api"
Oct 30 21:24:51.644725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.644725    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52aedae1-e213-4c88-bf1c-d21d4d563208" containerName="maas-api"
Oct 30 21:24:51.644982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.644967    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="52aedae1-e213-4c88-bf1c-d21d4d563208" containerName="maas-api"
Oct 30 21:24:51.650795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.650772    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"]
Oct 30 21:24:51.650934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.650910    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:51.660034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.659966    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-67b92c\""
Oct 30 21:24:51.660234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.660217    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-67b92c-dockercfg-gmhvx\""
Oct 30 21:24:51.672038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.672014    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"]
Oct 30 21:24:51.676654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.676631    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:51.692926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.692901    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"]
Oct 30 21:24:51.787695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.787188    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cdaf41-3928-4474-b8eb-157c5743b20b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd\" (UID: \"08cdaf41-3928-4474-b8eb-157c5743b20b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:51.787695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.787251    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjzq\" (UniqueName: \"kubernetes.io/projected/08cdaf41-3928-4474-b8eb-157c5743b20b-kube-api-access-sxjzq\") pod \"payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd\" (UID: \"08cdaf41-3928-4474-b8eb-157c5743b20b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:51.787695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.787329    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ec41d8-4028-4969-920f-3feeb852d494-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9\" (UID: \"82ec41d8-4028-4969-920f-3feeb852d494\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:51.787695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.787458    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggflp\" (UniqueName: \"kubernetes.io/projected/82ec41d8-4028-4969-920f-3feeb852d494-kube-api-access-ggflp\") pod \"payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9\" (UID: \"82ec41d8-4028-4969-920f-3feeb852d494\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:51.954750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.923942    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cdaf41-3928-4474-b8eb-157c5743b20b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd\" (UID: \"08cdaf41-3928-4474-b8eb-157c5743b20b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:51.954750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.924015    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjzq\" (UniqueName: \"kubernetes.io/projected/08cdaf41-3928-4474-b8eb-157c5743b20b-kube-api-access-sxjzq\") pod \"payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd\" (UID: \"08cdaf41-3928-4474-b8eb-157c5743b20b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:51.954750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.924108    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ec41d8-4028-4969-920f-3feeb852d494-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9\" (UID: \"82ec41d8-4028-4969-920f-3feeb852d494\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:51.954750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.924277    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggflp\" (UniqueName: \"kubernetes.io/projected/82ec41d8-4028-4969-920f-3feeb852d494-kube-api-access-ggflp\") pod \"payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9\" (UID: \"82ec41d8-4028-4969-920f-3feeb852d494\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:51.954750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.925969    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cdaf41-3928-4474-b8eb-157c5743b20b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd\" (UID: \"08cdaf41-3928-4474-b8eb-157c5743b20b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:51.956078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:51.955963    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ec41d8-4028-4969-920f-3feeb852d494-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9\" (UID: \"82ec41d8-4028-4969-920f-3feeb852d494\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:52.009887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.009854    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:24:52.010340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.010294    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 21:24:52.056932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.056531    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjzq\" (UniqueName: \"kubernetes.io/projected/08cdaf41-3928-4474-b8eb-157c5743b20b-kube-api-access-sxjzq\") pod \"payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd\" (UID: \"08cdaf41-3928-4474-b8eb-157c5743b20b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:52.065401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.065320    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggflp\" (UniqueName: \"kubernetes.io/projected/82ec41d8-4028-4969-920f-3feeb852d494-kube-api-access-ggflp\") pod \"payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9\" (UID: \"82ec41d8-4028-4969-920f-3feeb852d494\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:52.068409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.068386    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"]
Oct 30 21:24:52.072770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.072746    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-18b363-6cfb8df574-ckjn5"]
Oct 30 21:24:52.269291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.269188    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:52.298094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.298057    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:52.744352 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:24:52.744320    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ec41d8_4028_4969_920f_3feeb852d494.slice/crio-127bbb9b9265bac92f02e8aec17101e6a9d42156de877324d5b3619000e19353 WatchSource:0}: Error finding container 127bbb9b9265bac92f02e8aec17101e6a9d42156de877324d5b3619000e19353: Status 404 returned error can't find the container with id 127bbb9b9265bac92f02e8aec17101e6a9d42156de877324d5b3619000e19353
Oct 30 21:24:52.744574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.744556    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"]
Oct 30 21:24:52.786261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.786239    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"]
Oct 30 21:24:52.790015 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:24:52.789989    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08cdaf41_3928_4474_b8eb_157c5743b20b.slice/crio-6321a434fcfe578d50d55da1e4ac01f81c95d5e48f68b7ca7bbea20604879059 WatchSource:0}: Error finding container 6321a434fcfe578d50d55da1e4ac01f81c95d5e48f68b7ca7bbea20604879059: Status 404 returned error can't find the container with id 6321a434fcfe578d50d55da1e4ac01f81c95d5e48f68b7ca7bbea20604879059
Oct 30 21:24:52.809252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.809221    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:24:52.809374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:52.809273    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 21:24:53.540786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.540741    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9" event={"ID":"82ec41d8-4028-4969-920f-3feeb852d494","Type":"ContainerStarted","Data":"8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339"}
Oct 30 21:24:53.541300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.540812    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9" event={"ID":"82ec41d8-4028-4969-920f-3feeb852d494","Type":"ContainerStarted","Data":"127bbb9b9265bac92f02e8aec17101e6a9d42156de877324d5b3619000e19353"}
Oct 30 21:24:53.541300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.540841    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:24:53.542764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.542734    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd" event={"ID":"08cdaf41-3928-4474-b8eb-157c5743b20b","Type":"ContainerStarted","Data":"2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4"}
Oct 30 21:24:53.542886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.542786    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd" event={"ID":"08cdaf41-3928-4474-b8eb-157c5743b20b","Type":"ContainerStarted","Data":"6321a434fcfe578d50d55da1e4ac01f81c95d5e48f68b7ca7bbea20604879059"}
Oct 30 21:24:53.542886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.542808    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:24:53.559164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.559117    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9" podStartSLOduration=2.5591044 podStartE2EDuration="2.5591044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:53.556808959 +0000 UTC m=+1401.906536355" watchObservedRunningTime="2026-10-30 21:24:53.5591044 +0000 UTC m=+1401.908831796"
Oct 30 21:24:53.576096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.576049    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd" podStartSLOduration=2.576032607 podStartE2EDuration="2.576032607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:53.573326475 +0000 UTC m=+1401.923053873" watchObservedRunningTime="2026-10-30 21:24:53.576032607 +0000 UTC m=+1401.925760008"
Oct 30 21:24:53.805623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.805538    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52aedae1-e213-4c88-bf1c-d21d4d563208" path="/var/lib/kubelet/pods/52aedae1-e213-4c88-bf1c-d21d4d563208/volumes"
Oct 30 21:24:53.809152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.809122    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:24:53.809273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:53.809187    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 21:24:54.809295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:54.809262    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 21:24:54.809730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:54.809317    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 21:24:55.018430 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:24:55.018381    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:24:56.129131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:56.129096    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:56.129613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:56.129488    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:56.130257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:56.130241    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:24:58.756358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:58.756321    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"]
Oct 30 21:24:58.757540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:58.756556    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy" containerID="cri-o://83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2" gracePeriod=30
Oct 30 21:24:59.472635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:24:59.472605    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:25:02.619894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:02.619856    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:25:03.086208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:03.086174    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:25:03.086476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:03.086448    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:25:04.183404 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.183375    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:25:04.318307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318206    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10996acb-8603-4451-804e-d808664017d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10996acb-8603-4451-804e-d808664017d5-istiod-ca-cert\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.318307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318298    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-istio-token\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-istio-token\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.318554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318383    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-credential-socket\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.318554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318417    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10996acb-8603-4451-804e-d808664017d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10996acb-8603-4451-804e-d808664017d5-istio-podinfo\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.318554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318476    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-socket\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.318554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318504    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-kube-api-access-g4gcz\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-kube-api-access-g4gcz\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.318554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318553    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-certs\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.318871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318745    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-data\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.318871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.318810    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-envoy\") pod \"10996acb-8603-4451-804e-d808664017d5\" (UID: \"10996acb-8603-4451-804e-d808664017d5\") "
Oct 30 21:25:04.321572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.321070    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-credential-socket" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:04.321572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.321277    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-socket" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:04.321572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.321436    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10996acb-8603-4451-804e-d808664017d5-istiod-ca-cert" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:04.321572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.321498    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-certs" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:04.321572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.321548    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-data" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:04.335952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.335893    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-kube-api-access-g4gcz" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "kube-api-access-g4gcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:04.338268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.338228    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10996acb-8603-4451-804e-d808664017d5-istio-podinfo" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:04.339167 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.339128    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-envoy" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:04.339683 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.339644    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-istio-token" pod "10996acb-8603-4451-804e-d808664017d5" (UID: "10996acb-8603-4451-804e-d808664017d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.420862    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.420904    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.420921    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10996acb-8603-4451-804e-d808664017d5-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.420935    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.420948    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.420960    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10996acb-8603-4451-804e-d808664017d5-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.420974    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.420987    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4gcz\" (UniqueName: \"kubernetes.io/projected/10996acb-8603-4451-804e-d808664017d5-kube-api-access-g4gcz\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.427690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.421003    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10996acb-8603-4451-804e-d808664017d5-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:04.572611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.570899    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:25:04.590858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.573807    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:25:04.632461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.615503    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-nb8q8_d3af9a3e-eac9-4673-b9da-fc0dc10bab32/payload-processing/0.log"
Oct 30 21:25:04.884056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.883207    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-nb8q8_d3af9a3e-eac9-4673-b9da-fc0dc10bab32/payload-processing/0.log"
Oct 30 21:25:04.989714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.946207    4633 generic.go:356] "Generic (PLEG): container finished" podID="10996acb-8603-4451-804e-d808664017d5" containerID="83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2" exitCode=0
Oct 30 21:25:04.989714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.946546    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" event={"ID":"10996acb-8603-4451-804e-d808664017d5","Type":"ContainerDied","Data":"83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2"}
Oct 30 21:25:04.989714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.946607    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5" event={"ID":"10996acb-8603-4451-804e-d808664017d5","Type":"ContainerDied","Data":"0528fc81d253318575f6a7e0451f3f14c89900f46fde84e6c0b97eb9b9730de2"}
Oct 30 21:25:04.989714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.946643    4633 scope.go:122] "RemoveContainer" containerID="83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2"
Oct 30 21:25:04.989714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:04.946874    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"
Oct 30 21:25:05.506548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:05.506459    4633 scope.go:122] "RemoveContainer" containerID="80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984"
Oct 30 21:25:05.608691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:05.606761    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"]
Oct 30 21:25:05.629215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:05.620627    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-18b363-openshift-default-b5887d88f-4pkz5"]
Oct 30 21:25:05.722707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:05.722682    4633 scope.go:122] "RemoveContainer" containerID="83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2"
Oct 30 21:25:05.723541 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:05.723500    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2\": container with ID starting with 83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2 not found: ID does not exist" containerID="83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2"
Oct 30 21:25:05.724139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:05.724106    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2"} err="failed to get container status \"83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2\": rpc error: code = NotFound desc = could not find container \"83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2\": container with ID starting with 83905db0c167af7d93f0d4ead485ae2638efc35ec325ca0c4db42f0cb97cced2 not found: ID does not exist"
Oct 30 21:25:05.724307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:05.724291    4633 scope.go:122] "RemoveContainer" containerID="80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984"
Oct 30 21:25:05.724893 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:05.724871    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984\": container with ID starting with 80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984 not found: ID does not exist" containerID="80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984"
Oct 30 21:25:05.725163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:05.725119    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984"} err="failed to get container status \"80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984\": rpc error: code = NotFound desc = could not find container \"80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984\": container with ID starting with 80f21085c6fe7fb3ef7cf26559692da817a75b8caaaaeada1768003343e65984 not found: ID does not exist"
Oct 30 21:25:05.827837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:05.827772    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10996acb-8603-4451-804e-d808664017d5" path="/var/lib/kubelet/pods/10996acb-8603-4451-804e-d808664017d5/volumes"
Oct 30 21:25:15.275465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:15.275431    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"]
Oct 30 21:25:15.275943 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:15.275635    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy" containerID="cri-o://854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53" gracePeriod=30
Oct 30 21:25:20.619323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.619298    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:25:20.623940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.623918    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-data\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.623953    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-envoy\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.623995    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-certs\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624072    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-podinfo\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624155    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-kube-api-access-qvmxn\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-kube-api-access-qvmxn\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624192    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-data" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:20.624228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624208    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-credential-socket\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624337    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-socket\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624405    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-certs" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:20.624487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624424    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istiod-ca-cert\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624449    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-credential-socket" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:20.624587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624517    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-token\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-token\") pod \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\" (UID: \"7891f3fc-84c5-4a05-a212-d58c22f26e5a\") "
Oct 30 21:25:20.624587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.624578    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-socket" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:20.625065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.625033    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istiod-ca-cert" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:20.625332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.625306    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:20.625332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.625337    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:20.625332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.625355    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:20.625332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.625368    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:20.625653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.625381    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:20.626575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.626550    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-envoy" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:20.626787 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.626771    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-token" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:20.627186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.627162    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-podinfo" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:20.627297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.627282    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-kube-api-access-qvmxn" pod "7891f3fc-84c5-4a05-a212-d58c22f26e5a" (UID: "7891f3fc-84c5-4a05-a212-d58c22f26e5a"). InnerVolumeSpecName "kube-api-access-qvmxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:20.727141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.727099    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:20.727141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.727132    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:20.727141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.727142    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmxn\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-kube-api-access-qvmxn\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:20.727398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:20.727152    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7891f3fc-84c5-4a05-a212-d58c22f26e5a-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:21.292339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.292306    4633 generic.go:356] "Generic (PLEG): container finished" podID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerID="854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53" exitCode=0
Oct 30 21:25:21.292504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.292356    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"
Oct 30 21:25:21.292504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.292397    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" event={"ID":"7891f3fc-84c5-4a05-a212-d58c22f26e5a","Type":"ContainerDied","Data":"854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53"}
Oct 30 21:25:21.292504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.292457    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn" event={"ID":"7891f3fc-84c5-4a05-a212-d58c22f26e5a","Type":"ContainerDied","Data":"1ca313f49105733cf6cdcaaf6777cb7741c68787e781bad109ffee8f467b5ece"}
Oct 30 21:25:21.292504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.292487    4633 scope.go:122] "RemoveContainer" containerID="854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53"
Oct 30 21:25:21.304947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.304931    4633 scope.go:122] "RemoveContainer" containerID="854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53"
Oct 30 21:25:21.305251 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:21.305230    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53\": container with ID starting with 854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53 not found: ID does not exist" containerID="854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53"
Oct 30 21:25:21.305296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.305260    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53"} err="failed to get container status \"854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53\": rpc error: code = NotFound desc = could not find container \"854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53\": container with ID starting with 854db99e79d5d645e6cd7fbfdeca17cdfad1e5977adfe0b54b516f15f2777b53 not found: ID does not exist"
Oct 30 21:25:21.317414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.317388    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"]
Oct 30 21:25:21.321625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.321604    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-md8nn"]
Oct 30 21:25:21.603659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.603561    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"]
Oct 30 21:25:21.643731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.643660    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"]
Oct 30 21:25:21.792277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:21.792232    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" path="/var/lib/kubelet/pods/7891f3fc-84c5-4a05-a212-d58c22f26e5a/volumes"
Oct 30 21:25:23.343210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.343173    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"]
Oct 30 21:25:23.343740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.343512    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg" podUID="2c360e20-b181-404b-988c-ea360ee107e9" containerName="maas-api" containerID="cri-o://707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da" gracePeriod=30
Oct 30 21:25:23.456960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.456920    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"]
Oct 30 21:25:23.457197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.457174    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb" podUID="dde1ca2c-3c92-447d-881e-443fffc4d519" containerName="payload-pre-processing" containerID="cri-o://0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf" gracePeriod=30
Oct 30 21:25:23.777707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.758881    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:25:23.807881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.807426    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-maas-api-tls\") pod \"2c360e20-b181-404b-988c-ea360ee107e9\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") "
Oct 30 21:25:23.807881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.807479    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-metrics-certs\") pod \"2c360e20-b181-404b-988c-ea360ee107e9\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") "
Oct 30 21:25:23.807881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.807619    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c360e20-b181-404b-988c-ea360ee107e9-kube-api-access-pxmsx\" (UniqueName: \"kubernetes.io/projected/2c360e20-b181-404b-988c-ea360ee107e9-kube-api-access-pxmsx\") pod \"2c360e20-b181-404b-988c-ea360ee107e9\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") "
Oct 30 21:25:23.807881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.807724    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c360e20-b181-404b-988c-ea360ee107e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c360e20-b181-404b-988c-ea360ee107e9-openshift-service-ca\") pod \"2c360e20-b181-404b-988c-ea360ee107e9\" (UID: \"2c360e20-b181-404b-988c-ea360ee107e9\") "
Oct 30 21:25:23.879319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.879264    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c360e20-b181-404b-988c-ea360ee107e9-openshift-service-ca" pod "2c360e20-b181-404b-988c-ea360ee107e9" (UID: "2c360e20-b181-404b-988c-ea360ee107e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:23.952401 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.952368    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c360e20-b181-404b-988c-ea360ee107e9-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:23.956480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.956440    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c360e20-b181-404b-988c-ea360ee107e9-kube-api-access-pxmsx" pod "2c360e20-b181-404b-988c-ea360ee107e9" (UID: "2c360e20-b181-404b-988c-ea360ee107e9"). InnerVolumeSpecName "kube-api-access-pxmsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:23.959187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.959153    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-metrics-certs" pod "2c360e20-b181-404b-988c-ea360ee107e9" (UID: "2c360e20-b181-404b-988c-ea360ee107e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:23.967725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.967690    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-maas-api-tls" pod "2c360e20-b181-404b-988c-ea360ee107e9" (UID: "2c360e20-b181-404b-988c-ea360ee107e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:23.982144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:23.982118    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:25:24.085693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.078290    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde1ca2c-3c92-447d-881e-443fffc4d519-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde1ca2c-3c92-447d-881e-443fffc4d519-plugins-config-volume\") pod \"dde1ca2c-3c92-447d-881e-443fffc4d519\" (UID: \"dde1ca2c-3c92-447d-881e-443fffc4d519\") "
Oct 30 21:25:24.085693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.078426    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde1ca2c-3c92-447d-881e-443fffc4d519-kube-api-access-46kkq\" (UniqueName: \"kubernetes.io/projected/dde1ca2c-3c92-447d-881e-443fffc4d519-kube-api-access-46kkq\") pod \"dde1ca2c-3c92-447d-881e-443fffc4d519\" (UID: \"dde1ca2c-3c92-447d-881e-443fffc4d519\") "
Oct 30 21:25:24.085693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.079045    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:24.085693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.079072    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c360e20-b181-404b-988c-ea360ee107e9-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:24.085693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.079117    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxmsx\" (UniqueName: \"kubernetes.io/projected/2c360e20-b181-404b-988c-ea360ee107e9-kube-api-access-pxmsx\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:24.097883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.097843    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde1ca2c-3c92-447d-881e-443fffc4d519-plugins-config-volume" pod "dde1ca2c-3c92-447d-881e-443fffc4d519" (UID: "dde1ca2c-3c92-447d-881e-443fffc4d519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:24.108455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.108407    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde1ca2c-3c92-447d-881e-443fffc4d519-kube-api-access-46kkq" pod "dde1ca2c-3c92-447d-881e-443fffc4d519" (UID: "dde1ca2c-3c92-447d-881e-443fffc4d519"). InnerVolumeSpecName "kube-api-access-46kkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:24.180359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.180260    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46kkq\" (UniqueName: \"kubernetes.io/projected/dde1ca2c-3c92-447d-881e-443fffc4d519-kube-api-access-46kkq\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:24.180359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.180292    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde1ca2c-3c92-447d-881e-443fffc4d519-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:24.320658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.320620    4633 generic.go:356] "Generic (PLEG): container finished" podID="dde1ca2c-3c92-447d-881e-443fffc4d519" containerID="0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf" exitCode=0
Oct 30 21:25:24.320658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.320685    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"
Oct 30 21:25:24.320961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.320705    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb" event={"ID":"dde1ca2c-3c92-447d-881e-443fffc4d519","Type":"ContainerDied","Data":"0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf"}
Oct 30 21:25:24.320961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.320752    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb" event={"ID":"dde1ca2c-3c92-447d-881e-443fffc4d519","Type":"ContainerDied","Data":"48b4ae8c14e5954d8dbd4bf71c97bfdb49b0383f2ea526d54174928a74ebb8a2"}
Oct 30 21:25:24.320961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.320776    4633 scope.go:122] "RemoveContainer" containerID="0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf"
Oct 30 21:25:24.322459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.322418    4633 generic.go:356] "Generic (PLEG): container finished" podID="2c360e20-b181-404b-988c-ea360ee107e9" containerID="707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da" exitCode=0
Oct 30 21:25:24.322459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.322448    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg" event={"ID":"2c360e20-b181-404b-988c-ea360ee107e9","Type":"ContainerDied","Data":"707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da"}
Oct 30 21:25:24.322623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.322497    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"
Oct 30 21:25:24.322623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.322501    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg" event={"ID":"2c360e20-b181-404b-988c-ea360ee107e9","Type":"ContainerDied","Data":"dfce9b6e741345b8811e1bdbc108fc5447fa15fa0d08e9c526716d35c2157f59"}
Oct 30 21:25:24.333601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.333586    4633 scope.go:122] "RemoveContainer" containerID="0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf"
Oct 30 21:25:24.333899 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:24.333878    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf\": container with ID starting with 0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf not found: ID does not exist" containerID="0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf"
Oct 30 21:25:24.333970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.333906    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf"} err="failed to get container status \"0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf\": rpc error: code = NotFound desc = could not find container \"0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf\": container with ID starting with 0b7bd48640d40ef94116ba61190ee5391f87b3b3683dafebd429d185172609cf not found: ID does not exist"
Oct 30 21:25:24.333970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.333922    4633 scope.go:122] "RemoveContainer" containerID="707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da"
Oct 30 21:25:24.343812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.343793    4633 scope.go:122] "RemoveContainer" containerID="707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da"
Oct 30 21:25:24.344108 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:24.344059    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da\": container with ID starting with 707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da not found: ID does not exist" containerID="707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da"
Oct 30 21:25:24.344108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.344081    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da"} err="failed to get container status \"707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da\": rpc error: code = NotFound desc = could not find container \"707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da\": container with ID starting with 707b524e7515260fc88e90132dea7822d928d7a2c5ec3e18bd358a88ffd881da not found: ID does not exist"
Oct 30 21:25:24.352607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.352585    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"]
Oct 30 21:25:24.357169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.357150    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0fc028-64b5fb74bc-xxbcb"]
Oct 30 21:25:24.380454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.378137    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"]
Oct 30 21:25:24.384470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:24.384446    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0fc028-cfcc46c57-k65wg"]
Oct 30 21:25:25.024201 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:25.024165    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:25:25.805244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:25.805205    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c360e20-b181-404b-988c-ea360ee107e9" path="/var/lib/kubelet/pods/2c360e20-b181-404b-988c-ea360ee107e9/volumes"
Oct 30 21:25:25.805688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:25.805649    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde1ca2c-3c92-447d-881e-443fffc4d519" path="/var/lib/kubelet/pods/dde1ca2c-3c92-447d-881e-443fffc4d519/volumes"
Oct 30 21:25:26.604464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:26.604422    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm" podUID="1edf50f5-bdf9-4268-9b4d-228f49b1910d" containerName="payload-processing" containerID="cri-o://47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762" gracePeriod=25
Oct 30 21:25:26.644642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:26.644610    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2" podUID="495f00f7-d937-44d7-b15d-0cfa16359a69" containerName="payload-pre-processing" containerID="cri-o://a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a" gracePeriod=25
Oct 30 21:25:26.934843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:26.934818    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:25:26.984968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:26.984918    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:25:27.025162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.025125    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edf50f5-bdf9-4268-9b4d-228f49b1910d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edf50f5-bdf9-4268-9b4d-228f49b1910d-plugins-config-volume\") pod \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\" (UID: \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\") "
Oct 30 21:25:27.025403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.025223    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495f00f7-d937-44d7-b15d-0cfa16359a69-kube-api-access-xnnb7\" (UniqueName: \"kubernetes.io/projected/495f00f7-d937-44d7-b15d-0cfa16359a69-kube-api-access-xnnb7\") pod \"495f00f7-d937-44d7-b15d-0cfa16359a69\" (UID: \"495f00f7-d937-44d7-b15d-0cfa16359a69\") "
Oct 30 21:25:27.025403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.025330    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495f00f7-d937-44d7-b15d-0cfa16359a69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495f00f7-d937-44d7-b15d-0cfa16359a69-plugins-config-volume\") pod \"495f00f7-d937-44d7-b15d-0cfa16359a69\" (UID: \"495f00f7-d937-44d7-b15d-0cfa16359a69\") "
Oct 30 21:25:27.025593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.025494    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edf50f5-bdf9-4268-9b4d-228f49b1910d-kube-api-access-kfz6v\" (UniqueName: \"kubernetes.io/projected/1edf50f5-bdf9-4268-9b4d-228f49b1910d-kube-api-access-kfz6v\") pod \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\" (UID: \"1edf50f5-bdf9-4268-9b4d-228f49b1910d\") "
Oct 30 21:25:27.025593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.025536    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edf50f5-bdf9-4268-9b4d-228f49b1910d-plugins-config-volume" pod "1edf50f5-bdf9-4268-9b4d-228f49b1910d" (UID: "1edf50f5-bdf9-4268-9b4d-228f49b1910d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:27.025883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.025856    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495f00f7-d937-44d7-b15d-0cfa16359a69-plugins-config-volume" pod "495f00f7-d937-44d7-b15d-0cfa16359a69" (UID: "495f00f7-d937-44d7-b15d-0cfa16359a69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:27.026410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.026305    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edf50f5-bdf9-4268-9b4d-228f49b1910d-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:27.026526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.026410    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495f00f7-d937-44d7-b15d-0cfa16359a69-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:27.028119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.028093    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495f00f7-d937-44d7-b15d-0cfa16359a69-kube-api-access-xnnb7" pod "495f00f7-d937-44d7-b15d-0cfa16359a69" (UID: "495f00f7-d937-44d7-b15d-0cfa16359a69"). InnerVolumeSpecName "kube-api-access-xnnb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:27.028422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.028403    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edf50f5-bdf9-4268-9b4d-228f49b1910d-kube-api-access-kfz6v" pod "1edf50f5-bdf9-4268-9b4d-228f49b1910d" (UID: "1edf50f5-bdf9-4268-9b4d-228f49b1910d"). InnerVolumeSpecName "kube-api-access-kfz6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:27.145108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.144803    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnnb7\" (UniqueName: \"kubernetes.io/projected/495f00f7-d937-44d7-b15d-0cfa16359a69-kube-api-access-xnnb7\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:27.145108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.144842    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfz6v\" (UniqueName: \"kubernetes.io/projected/1edf50f5-bdf9-4268-9b4d-228f49b1910d-kube-api-access-kfz6v\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:27.349928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.349893    4633 generic.go:356] "Generic (PLEG): container finished" podID="495f00f7-d937-44d7-b15d-0cfa16359a69" containerID="a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a" exitCode=0
Oct 30 21:25:27.350086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.349944    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"
Oct 30 21:25:27.350086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.349978    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2" event={"ID":"495f00f7-d937-44d7-b15d-0cfa16359a69","Type":"ContainerDied","Data":"a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a"}
Oct 30 21:25:27.350086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.350034    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2" event={"ID":"495f00f7-d937-44d7-b15d-0cfa16359a69","Type":"ContainerDied","Data":"c33938384770a9e1f3f59e560f752274347e61b0773a49458c56a389987a6e1c"}
Oct 30 21:25:27.350086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.350067    4633 scope.go:122] "RemoveContainer" containerID="a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a"
Oct 30 21:25:27.351529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.351503    4633 generic.go:356] "Generic (PLEG): container finished" podID="1edf50f5-bdf9-4268-9b4d-228f49b1910d" containerID="47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762" exitCode=0
Oct 30 21:25:27.351663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.351616    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"
Oct 30 21:25:27.351789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.351662    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm" event={"ID":"1edf50f5-bdf9-4268-9b4d-228f49b1910d","Type":"ContainerDied","Data":"47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762"}
Oct 30 21:25:27.351789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.351747    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm" event={"ID":"1edf50f5-bdf9-4268-9b4d-228f49b1910d","Type":"ContainerDied","Data":"e819b75d337745760c922dac876cc394227d9e931c304ae8cc6e724ec90a0b56"}
Oct 30 21:25:27.379018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.378966    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"]
Oct 30 21:25:27.379345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.379327    4633 scope.go:122] "RemoveContainer" containerID="a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a"
Oct 30 21:25:27.379840 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:27.379811    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a\": container with ID starting with a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a not found: ID does not exist" containerID="a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a"
Oct 30 21:25:27.379935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.379855    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a"} err="failed to get container status \"a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a\": rpc error: code = NotFound desc = could not find container \"a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a\": container with ID starting with a9c626375189c0b68f38335c8711dfa96c8e654a0633e119951fb2b5282c7a2a not found: ID does not exist"
Oct 30 21:25:27.379935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.379889    4633 scope.go:122] "RemoveContainer" containerID="47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762"
Oct 30 21:25:27.383991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.383968    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7db332-589bd8cb87-g98g2"]
Oct 30 21:25:27.391804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.391782    4633 scope.go:122] "RemoveContainer" containerID="47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762"
Oct 30 21:25:27.392147 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:27.392128    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762\": container with ID starting with 47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762 not found: ID does not exist" containerID="47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762"
Oct 30 21:25:27.392201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.392156    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762"} err="failed to get container status \"47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762\": rpc error: code = NotFound desc = could not find container \"47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762\": container with ID starting with 47a9407aa338fcffbc7f6d96fab536849192ede082fd73b696193706ae836762 not found: ID does not exist"
Oct 30 21:25:27.394497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.394477    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"]
Oct 30 21:25:27.398385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.398365    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7db332-5fb6bb6cc-k8rnm"]
Oct 30 21:25:27.792979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.792946    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edf50f5-bdf9-4268-9b4d-228f49b1910d" path="/var/lib/kubelet/pods/1edf50f5-bdf9-4268-9b4d-228f49b1910d/volumes"
Oct 30 21:25:27.793372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:27.793358    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495f00f7-d937-44d7-b15d-0cfa16359a69" path="/var/lib/kubelet/pods/495f00f7-d937-44d7-b15d-0cfa16359a69/volumes"
Oct 30 21:25:32.595291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:32.595253    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"]
Oct 30 21:25:32.595876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:32.595516    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy" containerID="cri-o://709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d" gracePeriod=30
Oct 30 21:25:37.311609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:37.311572    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:25:37.944554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:37.944523    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113152    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-podinfo\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113271    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-certs\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113338    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-socket\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113613    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-kube-api-access-zrb88\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-kube-api-access-zrb88\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113653    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-token\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-token\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113706    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-data\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113779    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istiod-ca-cert\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113815    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-envoy\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.119768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.113869    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-credential-socket\") pod \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\" (UID: \"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b\") "
Oct 30 21:25:38.145753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.138599    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-credential-socket" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:38.145753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.139536    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istiod-ca-cert" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:38.145753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.139819    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-data" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:38.145753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.139994    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-socket" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:38.145753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.140241    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-certs" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:38.177027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.170891    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-token" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:38.177545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.177490    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-envoy" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:38.216019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.215859    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.216019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.215901    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.216019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.215916    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.216019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.215931    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.216019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.215945    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.216019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.215961    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.216019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.215975    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.219714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.216859    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-podinfo" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:38.227359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.220010    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-kube-api-access-zrb88" pod "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" (UID: "408e751a-2e1e-4a24-af5f-4dc2b8a7c72b"). InnerVolumeSpecName "kube-api-access-zrb88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:38.317697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.317182    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrb88\" (UniqueName: \"kubernetes.io/projected/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-kube-api-access-zrb88\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.317697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.317225    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:38.519751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.519171    4633 generic.go:356] "Generic (PLEG): container finished" podID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerID="709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d" exitCode=0
Oct 30 21:25:38.519751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.519226    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" event={"ID":"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b","Type":"ContainerDied","Data":"709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d"}
Oct 30 21:25:38.519751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.519271    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"
Oct 30 21:25:38.519751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.519296    4633 scope.go:122] "RemoveContainer" containerID="709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d"
Oct 30 21:25:38.519751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.519281    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55" event={"ID":"408e751a-2e1e-4a24-af5f-4dc2b8a7c72b","Type":"ContainerDied","Data":"922b9b9b66bf9263dbe1242fdc161a00b530ee71e8aead81f8db25b5b811a4ff"}
Oct 30 21:25:38.608914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.604901    4633 scope.go:122] "RemoveContainer" containerID="60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19"
Oct 30 21:25:38.624884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.622879    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"]
Oct 30 21:25:38.629710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.625327    4633 scope.go:122] "RemoveContainer" containerID="709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d"
Oct 30 21:25:38.629710 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:38.628026    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d\": container with ID starting with 709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d not found: ID does not exist" containerID="709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d"
Oct 30 21:25:38.629710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.628064    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d"} err="failed to get container status \"709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d\": rpc error: code = NotFound desc = could not find container \"709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d\": container with ID starting with 709477c2112484bb2f1340c70109a843652cbaaf797ed7c6c7a1008dd4f1239d not found: ID does not exist"
Oct 30 21:25:38.629710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.628088    4633 scope.go:122] "RemoveContainer" containerID="60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19"
Oct 30 21:25:38.629710 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:38.628433    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19\": container with ID starting with 60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19 not found: ID does not exist" containerID="60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19"
Oct 30 21:25:38.629710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.628460    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19"} err="failed to get container status \"60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19\": rpc error: code = NotFound desc = could not find container \"60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19\": container with ID starting with 60de9aacc93e6c4a7e672e7982b7361d27efd36d8d4f13a38180c9a06ff71c19 not found: ID does not exist"
Oct 30 21:25:38.632721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.632661    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0fc028-openshift-default-588d8dc6df-wcf55"]
Oct 30 21:25:38.857756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.855492    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"]
Oct 30 21:25:38.857756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:38.855769    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" containerID="cri-o://34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901" gracePeriod=30
Oct 30 21:25:39.839695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:39.839404    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" path="/var/lib/kubelet/pods/408e751a-2e1e-4a24-af5f-4dc2b8a7c72b/volumes"
Oct 30 21:25:40.284242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:40.284204    4633 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:25:40.284856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:40.284822    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:25:40.325490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:40.325421    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:25:40.326212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:40.326113    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:25:44.148940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.148864    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:25:44.258544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.258485    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-kube-api-access-qj7m5\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-kube-api-access-qj7m5\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.258773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.258625    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-podinfo\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.258773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.258695    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-certs\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.258773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.258751    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-token\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-token\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.259026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.258808    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-socket\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.259026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.258846    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-envoy\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.259026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.258900    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-data\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.259026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.258988    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istiod-ca-cert\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.259224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.259027    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-credential-socket\") pod \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\" (UID: \"020842ff-b53f-47ce-a40f-72e4e56f2c2c\") "
Oct 30 21:25:44.259277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.259239    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-data" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.259460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.259436    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-credential-socket" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.259460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.259450    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istiod-ca-cert" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:44.260879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.259597    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-certs" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.260879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.259686    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-socket" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.260879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.259974    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.260879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.259996    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.260879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.260012    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.260879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.260028    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.260879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.260043    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.261857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.261830    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-token" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:44.261965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.261921    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-envoy" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.262203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.262184    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-podinfo" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:44.268881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.268848    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-kube-api-access-qj7m5" pod "020842ff-b53f-47ce-a40f-72e4e56f2c2c" (UID: "020842ff-b53f-47ce-a40f-72e4e56f2c2c"). InnerVolumeSpecName "kube-api-access-qj7m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:44.361023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.360993    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj7m5\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-kube-api-access-qj7m5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.361023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.361021    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.361023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.361032    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.361263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.361042    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020842ff-b53f-47ce-a40f-72e4e56f2c2c-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.668325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.668287    4633 generic.go:356] "Generic (PLEG): container finished" podID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerID="34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901" exitCode=0
Oct 30 21:25:44.668491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.668428    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"
Oct 30 21:25:44.668580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.668424    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" event={"ID":"020842ff-b53f-47ce-a40f-72e4e56f2c2c","Type":"ContainerDied","Data":"34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901"}
Oct 30 21:25:44.668833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.668627    4633 scope.go:122] "RemoveContainer" containerID="34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901"
Oct 30 21:25:44.668911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.668883    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl" event={"ID":"020842ff-b53f-47ce-a40f-72e4e56f2c2c","Type":"ContainerDied","Data":"f7aaf1f5295aec4446a67ebc4220c35939953881664a74094a88b2e4a59d9a9b"}
Oct 30 21:25:44.688292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.688272    4633 scope.go:122] "RemoveContainer" containerID="62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78"
Oct 30 21:25:44.701449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.701421    4633 scope.go:122] "RemoveContainer" containerID="34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901"
Oct 30 21:25:44.701841 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:44.701821    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901\": container with ID starting with 34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901 not found: ID does not exist" containerID="34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901"
Oct 30 21:25:44.701920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.701851    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901"} err="failed to get container status \"34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901\": rpc error: code = NotFound desc = could not find container \"34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901\": container with ID starting with 34020f289faa9e075b9c77451e4433193affac622d820b6c939db04391eef901 not found: ID does not exist"
Oct 30 21:25:44.701920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.701871    4633 scope.go:122] "RemoveContainer" containerID="62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78"
Oct 30 21:25:44.702185 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:44.702156    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78\": container with ID starting with 62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78 not found: ID does not exist" containerID="62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78"
Oct 30 21:25:44.702185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.702187    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78"} err="failed to get container status \"62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78\": rpc error: code = NotFound desc = could not find container \"62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78\": container with ID starting with 62956c8f957f108a995eb38a81b38f6246887ebb03c166a04185863088e1ed78 not found: ID does not exist"
Oct 30 21:25:44.707574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.707541    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"]
Oct 30 21:25:44.712078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:44.712052    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-fzjtl"]
Oct 30 21:25:45.793020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:45.792986    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" path="/var/lib/kubelet/pods/020842ff-b53f-47ce-a40f-72e4e56f2c2c/volumes"
Oct 30 21:25:46.346037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:46.345997    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"]
Oct 30 21:25:46.363732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:46.363700    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"]
Oct 30 21:25:49.101816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.101781    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"]
Oct 30 21:25:49.102327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.102043    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp" podUID="0ca9dc71-d72d-456e-9c02-624e03cdc4ff" containerName="maas-api" containerID="cri-o://b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09" gracePeriod=30
Oct 30 21:25:49.197963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.197925    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"]
Oct 30 21:25:49.198213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.198174    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9" podUID="5c5ff786-1734-4d96-b69a-e0057ea851c7" containerName="payload-pre-processing" containerID="cri-o://e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0" gracePeriod=30
Oct 30 21:25:49.545929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.545897    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:25:49.689055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.689030    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:25:49.704984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.704951    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-metrics-certs\") pod \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") "
Oct 30 21:25:49.705327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.705285    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-openshift-service-ca\") pod \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") "
Oct 30 21:25:49.706301 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.705387    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-kube-api-access-pzx6w\" (UniqueName: \"kubernetes.io/projected/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-kube-api-access-pzx6w\") pod \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") "
Oct 30 21:25:49.706301 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.705421    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls\") pod \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\" (UID: \"0ca9dc71-d72d-456e-9c02-624e03cdc4ff\") "
Oct 30 21:25:49.706983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.706954    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-openshift-service-ca" pod "0ca9dc71-d72d-456e-9c02-624e03cdc4ff" (UID: "0ca9dc71-d72d-456e-9c02-624e03cdc4ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:49.708453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.708384    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-metrics-certs" pod "0ca9dc71-d72d-456e-9c02-624e03cdc4ff" (UID: "0ca9dc71-d72d-456e-9c02-624e03cdc4ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:49.709414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.709386    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls" pod "0ca9dc71-d72d-456e-9c02-624e03cdc4ff" (UID: "0ca9dc71-d72d-456e-9c02-624e03cdc4ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:49.710255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.710227    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-kube-api-access-pzx6w" pod "0ca9dc71-d72d-456e-9c02-624e03cdc4ff" (UID: "0ca9dc71-d72d-456e-9c02-624e03cdc4ff"). InnerVolumeSpecName "kube-api-access-pzx6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:49.737567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.737527    4633 generic.go:356] "Generic (PLEG): container finished" podID="0ca9dc71-d72d-456e-9c02-624e03cdc4ff" containerID="b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09" exitCode=0
Oct 30 21:25:49.737801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.737590    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"
Oct 30 21:25:49.737801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.737620    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp" event={"ID":"0ca9dc71-d72d-456e-9c02-624e03cdc4ff","Type":"ContainerDied","Data":"b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09"}
Oct 30 21:25:49.737801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.737719    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp" event={"ID":"0ca9dc71-d72d-456e-9c02-624e03cdc4ff","Type":"ContainerDied","Data":"22fa1f3ca941723d4c9e74f7cd3836c5dd2509993f4144e229f8308cf9d021b9"}
Oct 30 21:25:49.737801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.737753    4633 scope.go:122] "RemoveContainer" containerID="b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09"
Oct 30 21:25:49.739578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.739555    4633 generic.go:356] "Generic (PLEG): container finished" podID="5c5ff786-1734-4d96-b69a-e0057ea851c7" containerID="e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0" exitCode=0
Oct 30 21:25:49.739738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.739606    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"
Oct 30 21:25:49.739851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.739624    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9" event={"ID":"5c5ff786-1734-4d96-b69a-e0057ea851c7","Type":"ContainerDied","Data":"e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0"}
Oct 30 21:25:49.739926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.739883    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9" event={"ID":"5c5ff786-1734-4d96-b69a-e0057ea851c7","Type":"ContainerDied","Data":"5a8f39b81961eb7062d55de0a1b5675d4e2278675d8e41761d5767182b54a246"}
Oct 30 21:25:49.750830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.750807    4633 scope.go:122] "RemoveContainer" containerID="b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09"
Oct 30 21:25:49.751168 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:49.751149    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09\": container with ID starting with b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09 not found: ID does not exist" containerID="b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09"
Oct 30 21:25:49.751232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.751179    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09"} err="failed to get container status \"b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09\": rpc error: code = NotFound desc = could not find container \"b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09\": container with ID starting with b36e29a2e0540edde0a9e772f6e0526334a8c8912070f902ccae3f3918304e09 not found: ID does not exist"
Oct 30 21:25:49.751232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.751198    4633 scope.go:122] "RemoveContainer" containerID="e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0"
Oct 30 21:25:49.813403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.812095    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c5ff786-1734-4d96-b69a-e0057ea851c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5ff786-1734-4d96-b69a-e0057ea851c7-plugins-config-volume\") pod \"5c5ff786-1734-4d96-b69a-e0057ea851c7\" (UID: \"5c5ff786-1734-4d96-b69a-e0057ea851c7\") "
Oct 30 21:25:49.813403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.812254    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c5ff786-1734-4d96-b69a-e0057ea851c7-kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/5c5ff786-1734-4d96-b69a-e0057ea851c7-kube-api-access-9tf4f\") pod \"5c5ff786-1734-4d96-b69a-e0057ea851c7\" (UID: \"5c5ff786-1734-4d96-b69a-e0057ea851c7\") "
Oct 30 21:25:49.813403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.812941    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:49.813403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.812964    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzx6w\" (UniqueName: \"kubernetes.io/projected/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-kube-api-access-pzx6w\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:49.813403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.812980    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:49.813403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.812996    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ca9dc71-d72d-456e-9c02-624e03cdc4ff-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:49.824719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.824203    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5ff786-1734-4d96-b69a-e0057ea851c7-plugins-config-volume" pod "5c5ff786-1734-4d96-b69a-e0057ea851c7" (UID: "5c5ff786-1734-4d96-b69a-e0057ea851c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:49.824719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.824408    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5ff786-1734-4d96-b69a-e0057ea851c7-kube-api-access-9tf4f" pod "5c5ff786-1734-4d96-b69a-e0057ea851c7" (UID: "5c5ff786-1734-4d96-b69a-e0057ea851c7"). InnerVolumeSpecName "kube-api-access-9tf4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:49.826100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.825655    4633 scope.go:122] "RemoveContainer" containerID="e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0"
Oct 30 21:25:49.826100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.825805    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"]
Oct 30 21:25:49.838336 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:49.830108    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0\": container with ID starting with e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0 not found: ID does not exist" containerID="e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0"
Oct 30 21:25:49.838336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.830142    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0"} err="failed to get container status \"e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0\": rpc error: code = NotFound desc = could not find container \"e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0\": container with ID starting with e27ede413bb10ef959d4483794e934cbe76c3eb90a63c4b549bdf3a6367be2b0 not found: ID does not exist"
Oct 30 21:25:49.846829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.846799    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fde255-55f7cc5db-p49dp"]
Oct 30 21:25:49.914427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.914379    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c5ff786-1734-4d96-b69a-e0057ea851c7-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:49.914427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:49.914422    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tf4f\" (UniqueName: \"kubernetes.io/projected/5c5ff786-1734-4d96-b69a-e0057ea851c7-kube-api-access-9tf4f\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:50.058203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:50.058123    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"]
Oct 30 21:25:50.066534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:50.066503    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9"]
Oct 30 21:25:50.472558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:50.472515    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-5f5ff494c9-bfqc9" podUID="5c5ff786-1734-4d96-b69a-e0057ea851c7" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.167:9004: i/o timeout"
Oct 30 21:25:51.346341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.346301    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9" podUID="82ec41d8-4028-4969-920f-3feeb852d494" containerName="payload-processing" containerID="cri-o://8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339" gracePeriod=25
Oct 30 21:25:51.364550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.364510    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd" podUID="08cdaf41-3928-4474-b8eb-157c5743b20b" containerName="payload-pre-processing" containerID="cri-o://2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4" gracePeriod=25
Oct 30 21:25:51.659215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.659182    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:25:51.685008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.684974    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ec41d8-4028-4969-920f-3feeb852d494-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ec41d8-4028-4969-920f-3feeb852d494-plugins-config-volume\") pod \"82ec41d8-4028-4969-920f-3feeb852d494\" (UID: \"82ec41d8-4028-4969-920f-3feeb852d494\") "
Oct 30 21:25:51.685184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.685067    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ec41d8-4028-4969-920f-3feeb852d494-kube-api-access-ggflp\" (UniqueName: \"kubernetes.io/projected/82ec41d8-4028-4969-920f-3feeb852d494-kube-api-access-ggflp\") pod \"82ec41d8-4028-4969-920f-3feeb852d494\" (UID: \"82ec41d8-4028-4969-920f-3feeb852d494\") "
Oct 30 21:25:51.685435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.685407    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ec41d8-4028-4969-920f-3feeb852d494-plugins-config-volume" pod "82ec41d8-4028-4969-920f-3feeb852d494" (UID: "82ec41d8-4028-4969-920f-3feeb852d494"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:51.685701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.685645    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ec41d8-4028-4969-920f-3feeb852d494-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:51.687359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.687329    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ec41d8-4028-4969-920f-3feeb852d494-kube-api-access-ggflp" pod "82ec41d8-4028-4969-920f-3feeb852d494" (UID: "82ec41d8-4028-4969-920f-3feeb852d494"). InnerVolumeSpecName "kube-api-access-ggflp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:51.699464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.699440    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:25:51.846766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.846722    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca9dc71-d72d-456e-9c02-624e03cdc4ff" path="/var/lib/kubelet/pods/0ca9dc71-d72d-456e-9c02-624e03cdc4ff/volumes"
Oct 30 21:25:51.847437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.847411    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c5ff786-1734-4d96-b69a-e0057ea851c7" path="/var/lib/kubelet/pods/5c5ff786-1734-4d96-b69a-e0057ea851c7/volumes"
Oct 30 21:25:51.848016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.847989    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08cdaf41-3928-4474-b8eb-157c5743b20b-kube-api-access-sxjzq\" (UniqueName: \"kubernetes.io/projected/08cdaf41-3928-4474-b8eb-157c5743b20b-kube-api-access-sxjzq\") pod \"08cdaf41-3928-4474-b8eb-157c5743b20b\" (UID: \"08cdaf41-3928-4474-b8eb-157c5743b20b\") "
Oct 30 21:25:51.851878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.851850    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cdaf41-3928-4474-b8eb-157c5743b20b-kube-api-access-sxjzq" pod "08cdaf41-3928-4474-b8eb-157c5743b20b" (UID: "08cdaf41-3928-4474-b8eb-157c5743b20b"). InnerVolumeSpecName "kube-api-access-sxjzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:51.852611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.851913    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08cdaf41-3928-4474-b8eb-157c5743b20b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cdaf41-3928-4474-b8eb-157c5743b20b-plugins-config-volume\") pod \"08cdaf41-3928-4474-b8eb-157c5743b20b\" (UID: \"08cdaf41-3928-4474-b8eb-157c5743b20b\") "
Oct 30 21:25:51.852904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.852883    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cdaf41-3928-4474-b8eb-157c5743b20b-plugins-config-volume" pod "08cdaf41-3928-4474-b8eb-157c5743b20b" (UID: "08cdaf41-3928-4474-b8eb-157c5743b20b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:51.853051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.853032    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxjzq\" (UniqueName: \"kubernetes.io/projected/08cdaf41-3928-4474-b8eb-157c5743b20b-kube-api-access-sxjzq\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:51.856474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.856415    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggflp\" (UniqueName: \"kubernetes.io/projected/82ec41d8-4028-4969-920f-3feeb852d494-kube-api-access-ggflp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:51.857992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.857970    4633 generic.go:356] "Generic (PLEG): container finished" podID="08cdaf41-3928-4474-b8eb-157c5743b20b" containerID="2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4" exitCode=0
Oct 30 21:25:51.858155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.858140    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"
Oct 30 21:25:51.859779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.859744    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd" event={"ID":"08cdaf41-3928-4474-b8eb-157c5743b20b","Type":"ContainerDied","Data":"2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4"}
Oct 30 21:25:51.859940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.859926    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd" event={"ID":"08cdaf41-3928-4474-b8eb-157c5743b20b","Type":"ContainerDied","Data":"6321a434fcfe578d50d55da1e4ac01f81c95d5e48f68b7ca7bbea20604879059"}
Oct 30 21:25:51.860037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.859960    4633 scope.go:122] "RemoveContainer" containerID="2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4"
Oct 30 21:25:51.860425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.860400    4633 generic.go:356] "Generic (PLEG): container finished" podID="82ec41d8-4028-4969-920f-3feeb852d494" containerID="8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339" exitCode=0
Oct 30 21:25:51.860512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.860436    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9" event={"ID":"82ec41d8-4028-4969-920f-3feeb852d494","Type":"ContainerDied","Data":"8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339"}
Oct 30 21:25:51.860512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.860472    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9" event={"ID":"82ec41d8-4028-4969-920f-3feeb852d494","Type":"ContainerDied","Data":"127bbb9b9265bac92f02e8aec17101e6a9d42156de877324d5b3619000e19353"}
Oct 30 21:25:51.860623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.860541    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"
Oct 30 21:25:51.875991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.875971    4633 scope.go:122] "RemoveContainer" containerID="2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4"
Oct 30 21:25:51.876292 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:51.876271    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4\": container with ID starting with 2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4 not found: ID does not exist" containerID="2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4"
Oct 30 21:25:51.876369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.876307    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4"} err="failed to get container status \"2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4\": rpc error: code = NotFound desc = could not find container \"2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4\": container with ID starting with 2018e1de006b834c8c9f42f0ae1985fd7a0a6cf41154b12bffcb8b2629ca9fb4 not found: ID does not exist"
Oct 30 21:25:51.876369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.876329    4633 scope.go:122] "RemoveContainer" containerID="8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339"
Oct 30 21:25:51.886986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.886847    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"]
Oct 30 21:25:51.887075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.887028    4633 scope.go:122] "RemoveContainer" containerID="8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339"
Oct 30 21:25:51.887356 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:51.887330    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339\": container with ID starting with 8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339 not found: ID does not exist" containerID="8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339"
Oct 30 21:25:51.887434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.887356    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339"} err="failed to get container status \"8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339\": rpc error: code = NotFound desc = could not find container \"8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339\": container with ID starting with 8c37bf10b85c697326ea6b603dcbfdbe6218b61af74abf5696561be2c23b7339 not found: ID does not exist"
Oct 30 21:25:51.890751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.890655    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-67b92c-7759dcb987-x8fc9"]
Oct 30 21:25:51.902659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.902635    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"]
Oct 30 21:25:51.907386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.907363    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-67b92c-8b4844b9f-z2pxd"]
Oct 30 21:25:51.957934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:51.957894    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08cdaf41-3928-4474-b8eb-157c5743b20b-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:53.791649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:53.791616    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08cdaf41-3928-4474-b8eb-157c5743b20b" path="/var/lib/kubelet/pods/08cdaf41-3928-4474-b8eb-157c5743b20b/volumes"
Oct 30 21:25:53.792095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:53.792079    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ec41d8-4028-4969-920f-3feeb852d494" path="/var/lib/kubelet/pods/82ec41d8-4028-4969-920f-3feeb852d494/volumes"
Oct 30 21:25:54.268510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:54.268479    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"]
Oct 30 21:25:54.268740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:54.268713    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" containerID="cri-o://c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b" gracePeriod=30
Oct 30 21:25:54.357149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:54.357119    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"]
Oct 30 21:25:54.369661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:54.369630    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"]
Oct 30 21:25:54.544893 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:54.544814    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:25:54.544893 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:54.544883    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:25:55.015087 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:55.015053    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:25:59.049925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.049888    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"]
Oct 30 21:25:59.050767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.050710    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8" podUID="cb63490a-6a55-4e83-bdea-4418722d07cd" containerName="maas-api" containerID="cri-o://2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888" gracePeriod=30
Oct 30 21:25:59.464761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.455185    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw" podUID="e2699055-92f7-4648-b876-08fcf4de3f25" containerName="payload-pre-processing" containerID="cri-o://01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c" gracePeriod=25
Oct 30 21:25:59.464761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.457358    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76" podUID="bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" containerName="payload-processing" containerID="cri-o://359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c" gracePeriod=25
Oct 30 21:25:59.696242 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:25:59.696210    4633 logging.go:55] [core] [Channel #2326 SubChannel #2327]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.152:50052", ServerName: "10.133.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.152:50052: connect: connection refused"
Oct 30 21:25:59.699643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.699349    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:25:59.762894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.762857    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-metrics-certs\") pod \"cb63490a-6a55-4e83-bdea-4418722d07cd\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") "
Oct 30 21:25:59.763317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.763288    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb63490a-6a55-4e83-bdea-4418722d07cd-kube-api-access-2plsq\" (UniqueName: \"kubernetes.io/projected/cb63490a-6a55-4e83-bdea-4418722d07cd-kube-api-access-2plsq\") pod \"cb63490a-6a55-4e83-bdea-4418722d07cd\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") "
Oct 30 21:25:59.763505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.763483    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls\") pod \"cb63490a-6a55-4e83-bdea-4418722d07cd\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") "
Oct 30 21:25:59.763662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.763640    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb63490a-6a55-4e83-bdea-4418722d07cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63490a-6a55-4e83-bdea-4418722d07cd-openshift-service-ca\") pod \"cb63490a-6a55-4e83-bdea-4418722d07cd\" (UID: \"cb63490a-6a55-4e83-bdea-4418722d07cd\") "
Oct 30 21:25:59.764207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.764175    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb63490a-6a55-4e83-bdea-4418722d07cd-openshift-service-ca" pod "cb63490a-6a55-4e83-bdea-4418722d07cd" (UID: "cb63490a-6a55-4e83-bdea-4418722d07cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:59.764359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.764338    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63490a-6a55-4e83-bdea-4418722d07cd-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.766244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.766223    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-metrics-certs" pod "cb63490a-6a55-4e83-bdea-4418722d07cd" (UID: "cb63490a-6a55-4e83-bdea-4418722d07cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:59.766650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.766624    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls" pod "cb63490a-6a55-4e83-bdea-4418722d07cd" (UID: "cb63490a-6a55-4e83-bdea-4418722d07cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:59.769976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.769952    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb63490a-6a55-4e83-bdea-4418722d07cd-kube-api-access-2plsq" pod "cb63490a-6a55-4e83-bdea-4418722d07cd" (UID: "cb63490a-6a55-4e83-bdea-4418722d07cd"). InnerVolumeSpecName "kube-api-access-2plsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:59.799563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.799536    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:25:59.866132 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.866103    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.866330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.866311    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2plsq\" (UniqueName: \"kubernetes.io/projected/cb63490a-6a55-4e83-bdea-4418722d07cd-kube-api-access-2plsq\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.866407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.866335    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63490a-6a55-4e83-bdea-4418722d07cd-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.875480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.875449    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:25:59.884195 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.884166    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:25:59.908434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.908406    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:25:59.955402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.955310    4633 generic.go:356] "Generic (PLEG): container finished" podID="bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" containerID="359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c" exitCode=0
Oct 30 21:25:59.955402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.955371    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"
Oct 30 21:25:59.955633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.955397    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76" event={"ID":"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73","Type":"ContainerDied","Data":"359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c"}
Oct 30 21:25:59.955633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.955463    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76" event={"ID":"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73","Type":"ContainerDied","Data":"e59261dd28e227b3704ee874cff08c7e0de750ab48beb5649b4990d792ead2ee"}
Oct 30 21:25:59.955633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.955497    4633 scope.go:122] "RemoveContainer" containerID="359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c"
Oct 30 21:25:59.957043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.957016    4633 generic.go:356] "Generic (PLEG): container finished" podID="e2699055-92f7-4648-b876-08fcf4de3f25" containerID="01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c" exitCode=0
Oct 30 21:25:59.957224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.957065    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"
Oct 30 21:25:59.957224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.957126    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw" event={"ID":"e2699055-92f7-4648-b876-08fcf4de3f25","Type":"ContainerDied","Data":"01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c"}
Oct 30 21:25:59.957224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.957192    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw" event={"ID":"e2699055-92f7-4648-b876-08fcf4de3f25","Type":"ContainerDied","Data":"a1a26de49b0b9c9079651778cb694a3e9193478ec89582577ea17bfa12da8ad8"}
Oct 30 21:25:59.959286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.959265    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"
Oct 30 21:25:59.959503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.959482    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8" event={"ID":"cb63490a-6a55-4e83-bdea-4418722d07cd","Type":"ContainerDied","Data":"2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888"}
Oct 30 21:25:59.960216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.959357    4633 generic.go:356] "Generic (PLEG): container finished" podID="cb63490a-6a55-4e83-bdea-4418722d07cd" containerID="2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888" exitCode=0
Oct 30 21:25:59.960330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.960297    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8" event={"ID":"cb63490a-6a55-4e83-bdea-4418722d07cd","Type":"ContainerDied","Data":"7c4cd67ded228521c8c3ac2a5c8c54ce8e820d70786f9f93f4edf303ae42cc76"}
Oct 30 21:25:59.962055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.962021    4633 generic.go:356] "Generic (PLEG): container finished" podID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerID="c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b" exitCode=0
Oct 30 21:25:59.962055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.962050    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" event={"ID":"06f6f617-37a2-45c6-b9ad-662c12df9fb5","Type":"ContainerDied","Data":"c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b"}
Oct 30 21:25:59.962330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.962077    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"
Oct 30 21:25:59.962330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.962106    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl" event={"ID":"06f6f617-37a2-45c6-b9ad-662c12df9fb5","Type":"ContainerDied","Data":"245f5da2c19c979837eb4f7a27768392e8c989a65c8851ce9cd6cc55ff049d02"}
Oct 30 21:25:59.966896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.966869    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-kube-api-access-xskxh\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-kube-api-access-xskxh\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.966996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.966914    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-credential-socket\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.966996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.966940    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2699055-92f7-4648-b876-08fcf4de3f25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2699055-92f7-4648-b876-08fcf4de3f25-plugins-config-volume\") pod \"e2699055-92f7-4648-b876-08fcf4de3f25\" (UID: \"e2699055-92f7-4648-b876-08fcf4de3f25\") "
Oct 30 21:25:59.966996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.966980    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-plugins-config-volume\") pod \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\" (UID: \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\") "
Oct 30 21:25:59.967172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967038    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-envoy\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.967172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967081    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istiod-ca-cert\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.967172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967165    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-podinfo\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.967388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967194    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-kube-api-access-pnjvg\" (UniqueName: \"kubernetes.io/projected/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-kube-api-access-pnjvg\") pod \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\" (UID: \"bf82d1dd-75e3-4bc8-847d-40fd2ef35f73\") "
Oct 30 21:25:59.967388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967246    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-certs\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.967388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967276    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2699055-92f7-4648-b876-08fcf4de3f25-kube-api-access-vh5k6\" (UniqueName: \"kubernetes.io/projected/e2699055-92f7-4648-b876-08fcf4de3f25-kube-api-access-vh5k6\") pod \"e2699055-92f7-4648-b876-08fcf4de3f25\" (UID: \"e2699055-92f7-4648-b876-08fcf4de3f25\") "
Oct 30 21:25:59.967388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967379    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-socket\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.967588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967400    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-token\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-token\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.967588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967427    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-data\") pod \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\" (UID: \"06f6f617-37a2-45c6-b9ad-662c12df9fb5\") "
Oct 30 21:25:59.967588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.967527    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-plugins-config-volume" pod "bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" (UID: "bf82d1dd-75e3-4bc8-847d-40fd2ef35f73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:59.968084 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.968061    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2699055-92f7-4648-b876-08fcf4de3f25-plugins-config-volume" pod "e2699055-92f7-4648-b876-08fcf4de3f25" (UID: "e2699055-92f7-4648-b876-08fcf4de3f25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:59.968147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.968125    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:59.969315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.969286    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istiod-ca-cert" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:59.969812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.969768    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-credential-socket" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:59.969925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.969814    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-socket" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:59.970035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.969994    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-data" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:59.970196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.970173    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-certs" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:59.970627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.970602    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2699055-92f7-4648-b876-08fcf4de3f25-kube-api-access-vh5k6" pod "e2699055-92f7-4648-b876-08fcf4de3f25" (UID: "e2699055-92f7-4648-b876-08fcf4de3f25"). InnerVolumeSpecName "kube-api-access-vh5k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:59.971222 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.971197    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-kube-api-access-pnjvg" pod "bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" (UID: "bf82d1dd-75e3-4bc8-847d-40fd2ef35f73"). InnerVolumeSpecName "kube-api-access-pnjvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:59.971376 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.971358    4633 scope.go:122] "RemoveContainer" containerID="359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c"
Oct 30 21:25:59.971977 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:59.971935    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c\": container with ID starting with 359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c not found: ID does not exist" containerID="359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c"
Oct 30 21:25:59.972095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.971986    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c"} err="failed to get container status \"359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c\": rpc error: code = NotFound desc = could not find container \"359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c\": container with ID starting with 359f959718365b3aa767dac99e16b54e30b41bc2a87f67f94cb20a0205b7237c not found: ID does not exist"
Oct 30 21:25:59.972095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.972009    4633 scope.go:122] "RemoveContainer" containerID="01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c"
Oct 30 21:25:59.973994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.973974    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-token" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:59.974067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.973999    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-kube-api-access-xskxh" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "kube-api-access-xskxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:59.974130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.974109    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-podinfo" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:59.975384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.975362    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-envoy" pod "06f6f617-37a2-45c6-b9ad-662c12df9fb5" (UID: "06f6f617-37a2-45c6-b9ad-662c12df9fb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:59.983574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.983555    4633 scope.go:122] "RemoveContainer" containerID="01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c"
Oct 30 21:25:59.983903 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:59.983878    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c\": container with ID starting with 01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c not found: ID does not exist" containerID="01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c"
Oct 30 21:25:59.983982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.983907    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c"} err="failed to get container status \"01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c\": rpc error: code = NotFound desc = could not find container \"01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c\": container with ID starting with 01a5133b757f73012feca548d1076b941e6c2d4c31eff4e598b9efd0aea20a2c not found: ID does not exist"
Oct 30 21:25:59.983982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.983925    4633 scope.go:122] "RemoveContainer" containerID="2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888"
Oct 30 21:25:59.993361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.993336    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"]
Oct 30 21:25:59.996050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.996026    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-50d070-6bb6db6765-xcgg8"]
Oct 30 21:25:59.997786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.997768    4633 scope.go:122] "RemoveContainer" containerID="2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888"
Oct 30 21:25:59.998126 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:25:59.998101    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888\": container with ID starting with 2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888 not found: ID does not exist" containerID="2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888"
Oct 30 21:25:59.998197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.998135    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888"} err="failed to get container status \"2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888\": rpc error: code = NotFound desc = could not find container \"2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888\": container with ID starting with 2a776c321004798e4f866a06a3cd9d6d1a54b495bc64dc0e269de7ec400fc888 not found: ID does not exist"
Oct 30 21:25:59.998197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:25:59.998153    4633 scope.go:122] "RemoveContainer" containerID="c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b"
Oct 30 21:26:00.009524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.009502    4633 scope.go:122] "RemoveContainer" containerID="c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b"
Oct 30 21:26:00.009847 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:26:00.009828    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b\": container with ID starting with c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b not found: ID does not exist" containerID="c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b"
Oct 30 21:26:00.009922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.009856    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b"} err="failed to get container status \"c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b\": rpc error: code = NotFound desc = could not find container \"c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b\": container with ID starting with c1ea7c293255aa83b2636a73ce13e9d70e9fa8053325b783c83c1bc78973609b not found: ID does not exist"
Oct 30 21:26:00.069846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069813    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.069846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069842    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.069846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069853    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069868    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xskxh\" (UniqueName: \"kubernetes.io/projected/06f6f617-37a2-45c6-b9ad-662c12df9fb5-kube-api-access-xskxh\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069882    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069891    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2699055-92f7-4648-b876-08fcf4de3f25-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069899    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069907    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069917    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f6f617-37a2-45c6-b9ad-662c12df9fb5-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069926    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjvg\" (UniqueName: \"kubernetes.io/projected/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73-kube-api-access-pnjvg\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069934    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f6f617-37a2-45c6-b9ad-662c12df9fb5-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.070432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.069947    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh5k6\" (UniqueName: \"kubernetes.io/projected/e2699055-92f7-4648-b876-08fcf4de3f25-kube-api-access-vh5k6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:00.286906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.286869    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"]
Oct 30 21:26:00.294133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.294101    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76"]
Oct 30 21:26:00.308315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.308287    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"]
Oct 30 21:26:00.312417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.312390    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-50d070-9b4f4d967-28tjw"]
Oct 30 21:26:00.324261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.324228    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"]
Oct 30 21:26:00.329565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.329532    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-vjqjl"]
Oct 30 21:26:00.701715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:00.700182    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-50d070-5999666c69-sxp76" podUID="bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.152:50052\" within 1s: context deadline exceeded"
Oct 30 21:26:00.701715 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:00.700616    4633 logging.go:55] [core] [Channel #2326 SubChannel #2327]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.152:50052", ServerName: "10.133.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.152:50052: operation was canceled"
Oct 30 21:26:01.846570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:01.846524    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" path="/var/lib/kubelet/pods/06f6f617-37a2-45c6-b9ad-662c12df9fb5/volumes"
Oct 30 21:26:01.863927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:01.863885    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" path="/var/lib/kubelet/pods/bf82d1dd-75e3-4bc8-847d-40fd2ef35f73/volumes"
Oct 30 21:26:01.865130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:01.865096    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb63490a-6a55-4e83-bdea-4418722d07cd" path="/var/lib/kubelet/pods/cb63490a-6a55-4e83-bdea-4418722d07cd/volumes"
Oct 30 21:26:01.866094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:01.866059    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2699055-92f7-4648-b876-08fcf4de3f25" path="/var/lib/kubelet/pods/e2699055-92f7-4648-b876-08fcf4de3f25/volumes"
Oct 30 21:26:06.454734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:06.454700    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"]
Oct 30 21:26:06.455179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:06.454926    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" containerID="cri-o://4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c" gracePeriod=30
Oct 30 21:26:07.145331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:07.145285    4633 patch_prober.go:29] interesting pod/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:26:07.145545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:07.145351    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:26:09.333354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.333308    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"]
Oct 30 21:26:09.334203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334163    4633 cpu_manager.go:414] "containerMap: removing container" podUID="5c5ff786-1734-4d96-b69a-e0057ea851c7" containerName="payload-pre-processing"
Oct 30 21:26:09.334203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334200    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c5ff786-1734-4d96-b69a-e0057ea851c7" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334220    4633 cpu_manager.go:414] "containerMap: removing container" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334230    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334244    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0ca9dc71-d72d-456e-9c02-624e03cdc4ff" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334249    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca9dc71-d72d-456e-9c02-624e03cdc4ff" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334257    4633 cpu_manager.go:414] "containerMap: removing container" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334261    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334270    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1edf50f5-bdf9-4268-9b4d-228f49b1910d" containerName="payload-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334274    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edf50f5-bdf9-4268-9b4d-228f49b1910d" containerName="payload-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334286    4633 cpu_manager.go:414] "containerMap: removing container" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334290    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334297    4633 cpu_manager.go:414] "containerMap: removing container" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334301    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334307    4633 cpu_manager.go:414] "containerMap: removing container" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334311    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334322    4633 cpu_manager.go:414] "containerMap: removing container" podUID="495f00f7-d937-44d7-b15d-0cfa16359a69" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334327    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495f00f7-d937-44d7-b15d-0cfa16359a69" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334337    4633 cpu_manager.go:414] "containerMap: removing container" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334342    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334350    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2c360e20-b181-404b-988c-ea360ee107e9" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334354    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c360e20-b181-404b-988c-ea360ee107e9" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334370    4633 cpu_manager.go:414] "containerMap: removing container" podUID="bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" containerName="payload-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334375    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" containerName="payload-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334385    4633 cpu_manager.go:414] "containerMap: removing container" podUID="dde1ca2c-3c92-447d-881e-443fffc4d519" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334390    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde1ca2c-3c92-447d-881e-443fffc4d519" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334398    4633 cpu_manager.go:414] "containerMap: removing container" podUID="82ec41d8-4028-4969-920f-3feeb852d494" containerName="payload-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334402    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ec41d8-4028-4969-920f-3feeb852d494" containerName="payload-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334409    4633 cpu_manager.go:414] "containerMap: removing container" podUID="cb63490a-6a55-4e83-bdea-4418722d07cd" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334412    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb63490a-6a55-4e83-bdea-4418722d07cd" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334420    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e2699055-92f7-4648-b876-08fcf4de3f25" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334425    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2699055-92f7-4648-b876-08fcf4de3f25" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334431    4633 cpu_manager.go:414] "containerMap: removing container" podUID="08cdaf41-3928-4474-b8eb-157c5743b20b" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334437    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cdaf41-3928-4474-b8eb-157c5743b20b" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334534    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7891f3fc-84c5-4a05-a212-d58c22f26e5a" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334545    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334551    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334559    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="495f00f7-d937-44d7-b15d-0cfa16359a69" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334566    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="10996acb-8603-4451-804e-d808664017d5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334571    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c360e20-b181-404b-988c-ea360ee107e9" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334579    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf82d1dd-75e3-4bc8-847d-40fd2ef35f73" containerName="payload-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334585    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde1ca2c-3c92-447d-881e-443fffc4d519" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334591    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ec41d8-4028-4969-920f-3feeb852d494" containerName="payload-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334597    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334602    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb63490a-6a55-4e83-bdea-4418722d07cd" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334607    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2699055-92f7-4648-b876-08fcf4de3f25" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334612    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334616    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cdaf41-3928-4474-b8eb-157c5743b20b" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334622    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c5ff786-1734-4d96-b69a-e0057ea851c7" containerName="payload-pre-processing"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334627    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334633    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca9dc71-d72d-456e-9c02-624e03cdc4ff" containerName="maas-api"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334638    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f6f617-37a2-45c6-b9ad-662c12df9fb5" containerName="istio-proxy"
Oct 30 21:26:09.334776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.334644    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edf50f5-bdf9-4268-9b4d-228f49b1910d" containerName="payload-processing"
Oct 30 21:26:09.340010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.339994    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.344200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.344175    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:09.345049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.345025    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:26:09.346807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.346787    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"]
Oct 30 21:26:09.405193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.405156    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.405193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.405194    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lwp\" (UniqueName: \"kubernetes.io/projected/42235461-f97b-49c4-b280-15ab4ce97199-kube-api-access-f5lwp\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.405479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.405458    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.525693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.506760    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.525693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.506867    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.525693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.506910    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lwp\" (UniqueName: \"kubernetes.io/projected/42235461-f97b-49c4-b280-15ab4ce97199-kube-api-access-f5lwp\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.525693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.520119    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.525693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.520440    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.526601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.526571    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lwp\" (UniqueName: \"kubernetes.io/projected/42235461-f97b-49c4-b280-15ab4ce97199-kube-api-access-f5lwp\") pod \"maas-api-revoke-sub-0b4a41fe-fm854\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.658267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.658232    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:09.951290 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:09.951259    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"]
Oct 30 21:26:09.952174 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:09.952148    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42235461_f97b_49c4_b280_15ab4ce97199.slice/crio-c78a9021e5003003c157d041cdc358b01bb3604a0bd78900ecf8af3dbece79b1 WatchSource:0}: Error finding container c78a9021e5003003c157d041cdc358b01bb3604a0bd78900ecf8af3dbece79b1: Status 404 returned error can't find the container with id c78a9021e5003003c157d041cdc358b01bb3604a0bd78900ecf8af3dbece79b1
Oct 30 21:26:10.295632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.292757    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854" event={"ID":"42235461-f97b-49c4-b280-15ab4ce97199","Type":"ContainerStarted","Data":"3dd2779997b8ca9f640e0d5411cda304b0a762f251a51e06c9f7acb2379ac053"}
Oct 30 21:26:10.295632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.292832    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854" event={"ID":"42235461-f97b-49c4-b280-15ab4ce97199","Type":"ContainerStarted","Data":"c78a9021e5003003c157d041cdc358b01bb3604a0bd78900ecf8af3dbece79b1"}
Oct 30 21:26:10.328913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.328326    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854" podStartSLOduration=1.328303207 podStartE2EDuration="1.328303207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:10.314906514 +0000 UTC m=+1478.664633910" watchObservedRunningTime="2026-10-30 21:26:10.328303207 +0000 UTC m=+1478.678030605"
Oct 30 21:26:10.793959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.793920    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"]
Oct 30 21:26:10.795018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.794996    4633 cpu_manager.go:414] "containerMap: removing container" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy"
Oct 30 21:26:10.795082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.795023    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020842ff-b53f-47ce-a40f-72e4e56f2c2c" containerName="istio-proxy"
Oct 30 21:26:10.795082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.795061    4633 cpu_manager.go:414] "containerMap: removing container" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy"
Oct 30 21:26:10.795082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.795069    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408e751a-2e1e-4a24-af5f-4dc2b8a7c72b" containerName="istio-proxy"
Oct 30 21:26:10.799422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.799402    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.804135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.804058    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b30a42-openshift-default-dockercfg-qgz6d\""
Oct 30 21:26:10.818918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.815191    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"]
Oct 30 21:26:10.965085 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.938076    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-envoy\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.965962 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.965933    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175ae88-19d0-4c92-8f35-649ac11138ac-istiod-ca-cert\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.980696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.966252    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-podinfo\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.981072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.981031    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-data\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.981395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.981375    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrcg\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-kube-api-access-7xrcg\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.981930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.981909    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-token\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.982226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.982209    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.982525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.982506    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-credential-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:10.982721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:10.982703    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-certs\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.084657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.084564    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-envoy\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.084925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.084904    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175ae88-19d0-4c92-8f35-649ac11138ac-istiod-ca-cert\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.085032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.085021    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-podinfo\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.085116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.085107    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-data\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.085213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.085202    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrcg\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-kube-api-access-7xrcg\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.085377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.085364    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-token\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.085496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.085486    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.085629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.085615    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-credential-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.085779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.085766    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-certs\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.086281 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.086182    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-certs\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.087086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.087065    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175ae88-19d0-4c92-8f35-649ac11138ac-istiod-ca-cert\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.088358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.088324    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-data\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.088747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.088727    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.088991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.088949    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-credential-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.092533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.092518    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-podinfo\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.093160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.093145    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-envoy\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.097766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.097746    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrcg\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-kube-api-access-7xrcg\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.098207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.098184    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-token\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.144243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.144212    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:11.342272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.342170    4633 generic.go:356] "Generic (PLEG): container finished" podID="42235461-f97b-49c4-b280-15ab4ce97199" containerID="3dd2779997b8ca9f640e0d5411cda304b0a762f251a51e06c9f7acb2379ac053" exitCode=0
Oct 30 21:26:11.342449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.342270    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854" event={"ID":"42235461-f97b-49c4-b280-15ab4ce97199","Type":"ContainerDied","Data":"3dd2779997b8ca9f640e0d5411cda304b0a762f251a51e06c9f7acb2379ac053"}
Oct 30 21:26:11.458787 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.458759    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"]
Oct 30 21:26:11.459385 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:11.459358    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc175ae88_19d0_4c92_8f35_649ac11138ac.slice/crio-e53991b922e50df4d48d2dd6805ec3b99ed12ad030555d793a51cda965ce5011 WatchSource:0}: Error finding container e53991b922e50df4d48d2dd6805ec3b99ed12ad030555d793a51cda965ce5011: Status 404 returned error can't find the container with id e53991b922e50df4d48d2dd6805ec3b99ed12ad030555d793a51cda965ce5011
Oct 30 21:26:11.462056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.462026    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:11.462121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.462093    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:11.462158 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.462122    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:11.751091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.751066    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:26:11.812743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.812710    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-envoy\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.812781    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-data\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.812924    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c29883ff-2844-4d5f-b016-9e87407390fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29883ff-2844-4d5f-b016-9e87407390fc-istio-podinfo\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.812959    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-kube-api-access-4ss7m\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-kube-api-access-4ss7m\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.813009    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-istio-token\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-istio-token\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.813086    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-socket\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.813126    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-certs\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.813178    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-credential-socket\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.813212    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29883ff-2844-4d5f-b016-9e87407390fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29883ff-2844-4d5f-b016-9e87407390fc-istiod-ca-cert\") pod \"c29883ff-2844-4d5f-b016-9e87407390fc\" (UID: \"c29883ff-2844-4d5f-b016-9e87407390fc\") "
Oct 30 21:26:11.813218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.813215    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-data" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.813824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.813800    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.814352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.814058    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-certs" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.814352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.814316    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-credential-socket" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.814575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.814382    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-socket" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.814575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.814426    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29883ff-2844-4d5f-b016-9e87407390fc-istiod-ca-cert" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:11.817050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.816997    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-istio-token" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:11.817752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.817720    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c29883ff-2844-4d5f-b016-9e87407390fc-istio-podinfo" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:26:11.817856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.817809    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-envoy" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:11.817938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.817922    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-kube-api-access-4ss7m" pod "c29883ff-2844-4d5f-b016-9e87407390fc" (UID: "c29883ff-2844-4d5f-b016-9e87407390fc"). InnerVolumeSpecName "kube-api-access-4ss7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:11.915086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.915056    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29883ff-2844-4d5f-b016-9e87407390fc-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.915086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.915086    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ss7m\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-kube-api-access-4ss7m\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.915317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.915096    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29883ff-2844-4d5f-b016-9e87407390fc-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.915317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.915105    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.915317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.915115    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.915317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.915124    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.915317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.915133    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29883ff-2844-4d5f-b016-9e87407390fc-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.915317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:11.915141    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29883ff-2844-4d5f-b016-9e87407390fc-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:12.354782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.354626    4633 generic.go:356] "Generic (PLEG): container finished" podID="c29883ff-2844-4d5f-b016-9e87407390fc" containerID="4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c" exitCode=0
Oct 30 21:26:12.354782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.354709    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"
Oct 30 21:26:12.354782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.354724    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" event={"ID":"c29883ff-2844-4d5f-b016-9e87407390fc","Type":"ContainerDied","Data":"4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c"}
Oct 30 21:26:12.355127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.354803    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb" event={"ID":"c29883ff-2844-4d5f-b016-9e87407390fc","Type":"ContainerDied","Data":"2d3b12f3bda39667aa7d9b46f8b0c43a3f8d455c97e345bd280143f33ac52e9f"}
Oct 30 21:26:12.355127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.354832    4633 scope.go:122] "RemoveContainer" containerID="4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c"
Oct 30 21:26:12.361815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.361774    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" event={"ID":"c175ae88-19d0-4c92-8f35-649ac11138ac","Type":"ContainerStarted","Data":"939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0"}
Oct 30 21:26:12.361941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.361832    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" event={"ID":"c175ae88-19d0-4c92-8f35-649ac11138ac","Type":"ContainerStarted","Data":"e53991b922e50df4d48d2dd6805ec3b99ed12ad030555d793a51cda965ce5011"}
Oct 30 21:26:12.372844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.372813    4633 scope.go:122] "RemoveContainer" containerID="441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650"
Oct 30 21:26:12.394847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.394818    4633 scope.go:122] "RemoveContainer" containerID="4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c"
Oct 30 21:26:12.395377 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:26:12.395346    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c\": container with ID starting with 4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c not found: ID does not exist" containerID="4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c"
Oct 30 21:26:12.395510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.395383    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c"} err="failed to get container status \"4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c\": rpc error: code = NotFound desc = could not find container \"4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c\": container with ID starting with 4e63951908d92a179455ee7ca6eff616ccb987734e38e57c65a80e2d3952021c not found: ID does not exist"
Oct 30 21:26:12.395510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.395402    4633 scope.go:122] "RemoveContainer" containerID="441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650"
Oct 30 21:26:12.395736 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:26:12.395681    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650\": container with ID starting with 441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650 not found: ID does not exist" containerID="441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650"
Oct 30 21:26:12.395736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.395714    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650"} err="failed to get container status \"441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650\": rpc error: code = NotFound desc = could not find container \"441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650\": container with ID starting with 441f06f62e70b2ff4dc95cfe9bb2324145df9ae79b56dd8b50c8ea6a171e0650 not found: ID does not exist"
Oct 30 21:26:12.419307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.419251    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podStartSLOduration=2.419234861 podStartE2EDuration="2.419234861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:12.394369165 +0000 UTC m=+1480.744096575" watchObservedRunningTime="2026-10-30 21:26:12.419234861 +0000 UTC m=+1480.768962260"
Oct 30 21:26:12.423533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.423504    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"]
Oct 30 21:26:12.442624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.442592    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-50d070-openshift-default-76fcfcd8fc-cwswb"]
Oct 30 21:26:12.560429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.560400    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:12.657987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.657958    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42235461-f97b-49c4-b280-15ab4ce97199-kube-api-access-f5lwp\" (UniqueName: \"kubernetes.io/projected/42235461-f97b-49c4-b280-15ab4ce97199-kube-api-access-f5lwp\") pod \"42235461-f97b-49c4-b280-15ab4ce97199\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") "
Oct 30 21:26:12.658173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.658123    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-openshift-ca\") pod \"42235461-f97b-49c4-b280-15ab4ce97199\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") "
Oct 30 21:26:12.658173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.658155    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-xks-ca\") pod \"42235461-f97b-49c4-b280-15ab4ce97199\" (UID: \"42235461-f97b-49c4-b280-15ab4ce97199\") "
Oct 30 21:26:12.658514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.658487    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-openshift-ca" pod "42235461-f97b-49c4-b280-15ab4ce97199" (UID: "42235461-f97b-49c4-b280-15ab4ce97199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:12.658625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.658609    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-openshift-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:12.660375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.660353    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-xks-ca" pod "42235461-f97b-49c4-b280-15ab4ce97199" (UID: "42235461-f97b-49c4-b280-15ab4ce97199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:12.660375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.660363    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42235461-f97b-49c4-b280-15ab4ce97199-kube-api-access-f5lwp" pod "42235461-f97b-49c4-b280-15ab4ce97199" (UID: "42235461-f97b-49c4-b280-15ab4ce97199"). InnerVolumeSpecName "kube-api-access-f5lwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:12.759894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.759857    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42235461-f97b-49c4-b280-15ab4ce97199-maas-api-xks-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:12.759894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:12.759890    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lwp\" (UniqueName: \"kubernetes.io/projected/42235461-f97b-49c4-b280-15ab4ce97199-kube-api-access-f5lwp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:13.145469 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:13.145387    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:13.151150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:13.151121    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:13.371327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:13.371291    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"
Oct 30 21:26:13.371516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:13.371336    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854" event={"ID":"42235461-f97b-49c4-b280-15ab4ce97199","Type":"ContainerDied","Data":"c78a9021e5003003c157d041cdc358b01bb3604a0bd78900ecf8af3dbece79b1"}
Oct 30 21:26:13.371516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:13.371390    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c78a9021e5003003c157d041cdc358b01bb3604a0bd78900ecf8af3dbece79b1"
Oct 30 21:26:13.372886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:13.372863    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:13.373754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:13.373736    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:26:13.791485 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:13.791450    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" path="/var/lib/kubelet/pods/c29883ff-2844-4d5f-b016-9e87407390fc/volumes"
Oct 30 21:26:16.045299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.045263    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"]
Oct 30 21:26:16.045938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.045919    4633 cpu_manager.go:414] "containerMap: removing container" podUID="42235461-f97b-49c4-b280-15ab4ce97199" containerName="revoke-keys"
Oct 30 21:26:16.045938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.045938    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42235461-f97b-49c4-b280-15ab4ce97199" containerName="revoke-keys"
Oct 30 21:26:16.046086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.045956    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy"
Oct 30 21:26:16.046086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.045963    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy"
Oct 30 21:26:16.046086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.045974    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy"
Oct 30 21:26:16.046086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.045978    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy"
Oct 30 21:26:16.046086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.046085    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="42235461-f97b-49c4-b280-15ab4ce97199" containerName="revoke-keys"
Oct 30 21:26:16.046351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.046096    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy"
Oct 30 21:26:16.046351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.046104    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29883ff-2844-4d5f-b016-9e87407390fc" containerName="istio-proxy"
Oct 30 21:26:16.051016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.050989    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:16.062304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.062258    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-b30a42\""
Oct 30 21:26:16.062971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.062937    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b30a42-dockercfg-5l4bq\""
Oct 30 21:26:16.067806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.067772    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"]
Oct 30 21:26:16.073779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.073749    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"]
Oct 30 21:26:16.080972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.080942    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:16.090033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.089998    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b905e0bb-b295-4752-b8d1-19ae58096f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9\" (UID: \"b905e0bb-b295-4752-b8d1-19ae58096f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:16.090293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.090274    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p69f8\" (UniqueName: \"kubernetes.io/projected/b905e0bb-b295-4752-b8d1-19ae58096f46-kube-api-access-p69f8\") pod \"payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9\" (UID: \"b905e0bb-b295-4752-b8d1-19ae58096f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:16.090594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.090571    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"]
Oct 30 21:26:16.199853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.199810    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2rt\" (UniqueName: \"kubernetes.io/projected/05e11b5a-7968-4403-b6aa-2d83dcbff14b-kube-api-access-bj2rt\") pod \"payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d\" (UID: \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:16.200193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.200165    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e11b5a-7968-4403-b6aa-2d83dcbff14b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d\" (UID: \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:16.200559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.200529    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b905e0bb-b295-4752-b8d1-19ae58096f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9\" (UID: \"b905e0bb-b295-4752-b8d1-19ae58096f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:16.200876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.200838    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p69f8\" (UniqueName: \"kubernetes.io/projected/b905e0bb-b295-4752-b8d1-19ae58096f46-kube-api-access-p69f8\") pod \"payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9\" (UID: \"b905e0bb-b295-4752-b8d1-19ae58096f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:16.202290 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.202267    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b905e0bb-b295-4752-b8d1-19ae58096f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9\" (UID: \"b905e0bb-b295-4752-b8d1-19ae58096f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:16.237642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.237607    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p69f8\" (UniqueName: \"kubernetes.io/projected/b905e0bb-b295-4752-b8d1-19ae58096f46-kube-api-access-p69f8\") pod \"payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9\" (UID: \"b905e0bb-b295-4752-b8d1-19ae58096f46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:16.302750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.302567    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2rt\" (UniqueName: \"kubernetes.io/projected/05e11b5a-7968-4403-b6aa-2d83dcbff14b-kube-api-access-bj2rt\") pod \"payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d\" (UID: \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:16.302750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.302654    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e11b5a-7968-4403-b6aa-2d83dcbff14b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d\" (UID: \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:16.308696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.303865    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e11b5a-7968-4403-b6aa-2d83dcbff14b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d\" (UID: \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:16.330729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.329468    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2rt\" (UniqueName: \"kubernetes.io/projected/05e11b5a-7968-4403-b6aa-2d83dcbff14b-kube-api-access-bj2rt\") pod \"payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d\" (UID: \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:16.380794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.377852    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:16.406494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.406464    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:16.706689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.706631    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"]
Oct 30 21:26:16.706928 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:16.706905    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb905e0bb_b295_4752_b8d1_19ae58096f46.slice/crio-1f7ca7f3cac6db7da1b291c841724c6005a965756343d002e45b227a2746a79f WatchSource:0}: Error finding container 1f7ca7f3cac6db7da1b291c841724c6005a965756343d002e45b227a2746a79f: Status 404 returned error can't find the container with id 1f7ca7f3cac6db7da1b291c841724c6005a965756343d002e45b227a2746a79f
Oct 30 21:26:16.737012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:16.736986    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"]
Oct 30 21:26:16.743125 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:16.743097    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e11b5a_7968_4403_b6aa_2d83dcbff14b.slice/crio-73c2fbac08c131884280f4fd3206a6ccedc52b3833d233546fea8191d01e34c3 WatchSource:0}: Error finding container 73c2fbac08c131884280f4fd3206a6ccedc52b3833d233546fea8191d01e34c3: Status 404 returned error can't find the container with id 73c2fbac08c131884280f4fd3206a6ccedc52b3833d233546fea8191d01e34c3
Oct 30 21:26:17.416050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:17.415996    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d" event={"ID":"05e11b5a-7968-4403-b6aa-2d83dcbff14b","Type":"ContainerStarted","Data":"dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326"}
Oct 30 21:26:17.416520 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:17.416104    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d" event={"ID":"05e11b5a-7968-4403-b6aa-2d83dcbff14b","Type":"ContainerStarted","Data":"73c2fbac08c131884280f4fd3206a6ccedc52b3833d233546fea8191d01e34c3"}
Oct 30 21:26:17.416520 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:17.416133    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:17.417896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:17.417867    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9" event={"ID":"b905e0bb-b295-4752-b8d1-19ae58096f46","Type":"ContainerStarted","Data":"44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb"}
Oct 30 21:26:17.418041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:17.417912    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9" event={"ID":"b905e0bb-b295-4752-b8d1-19ae58096f46","Type":"ContainerStarted","Data":"1f7ca7f3cac6db7da1b291c841724c6005a965756343d002e45b227a2746a79f"}
Oct 30 21:26:17.418041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:17.418007    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:17.449214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:17.449155    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d" podStartSLOduration=1.449139278 podStartE2EDuration="1.449139278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:17.446590856 +0000 UTC m=+1485.796318257" watchObservedRunningTime="2026-10-30 21:26:17.449139278 +0000 UTC m=+1485.798866674"
Oct 30 21:26:17.471685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:17.471621    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9" podStartSLOduration=1.471607213 podStartE2EDuration="1.471607213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:17.471471005 +0000 UTC m=+1485.821198403" watchObservedRunningTime="2026-10-30 21:26:17.471607213 +0000 UTC m=+1485.821334609"
Oct 30 21:26:19.583295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.583256    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"]
Oct 30 21:26:19.588613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.588588    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.599578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.599547    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-b30a42-serving-cert\""
Oct 30 21:26:19.614946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.614917    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"]
Oct 30 21:26:19.647340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.647306    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.647565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.647366    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8cq\" (UniqueName: \"kubernetes.io/projected/ffd17c96-6623-4d56-b849-b4a94e75de97-kube-api-access-ln8cq\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.647628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.647550    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.647809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.647788    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd17c96-6623-4d56-b849-b4a94e75de97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.768695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.755710    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.769112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.765112    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.769423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.769401    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd17c96-6623-4d56-b849-b4a94e75de97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.786548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.783962    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.786548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.784033    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8cq\" (UniqueName: \"kubernetes.io/projected/ffd17c96-6623-4d56-b849-b4a94e75de97-kube-api-access-ln8cq\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.786548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.784444    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd17c96-6623-4d56-b849-b4a94e75de97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:19.786548 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:26:19.784501    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-serving-cert: secret "maas-api-e2e-shared-a-w0-b30a42-serving-cert" not found
Oct 30 21:26:19.786548 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:26:19.784557    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls podName:ffd17c96-6623-4d56-b849-b4a94e75de97 nodeName:}" failed. No retries permitted until 2026-10-30 21:26:20.284539296 +0000 UTC m=+1488.634266675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls") pod "maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz" (UID: "ffd17c96-6623-4d56-b849-b4a94e75de97") : secret "maas-api-e2e-shared-a-w0-b30a42-serving-cert" not found
Oct 30 21:26:19.800061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:19.798589    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8cq\" (UniqueName: \"kubernetes.io/projected/ffd17c96-6623-4d56-b849-b4a94e75de97-kube-api-access-ln8cq\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:20.297580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:20.297532    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:20.335508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:20.335393    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:20.513915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:20.513883    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:20.806866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:20.806836    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"]
Oct 30 21:26:20.807205 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:20.807008    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd17c96_6623_4d56_b849_b4a94e75de97.slice/crio-46ea2254f689c36bf73ce146d561d8ad44ae4a186340f45c945fce1b17c85f98 WatchSource:0}: Error finding container 46ea2254f689c36bf73ce146d561d8ad44ae4a186340f45c945fce1b17c85f98: Status 404 returned error can't find the container with id 46ea2254f689c36bf73ce146d561d8ad44ae4a186340f45c945fce1b17c85f98
Oct 30 21:26:21.294237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:21.294195    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:26:21.451532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:21.451490    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz" event={"ID":"ffd17c96-6623-4d56-b849-b4a94e75de97","Type":"ContainerStarted","Data":"4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0"}
Oct 30 21:26:21.451762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:21.451705    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:21.451847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:21.451751    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz" event={"ID":"ffd17c96-6623-4d56-b849-b4a94e75de97","Type":"ContainerStarted","Data":"46ea2254f689c36bf73ce146d561d8ad44ae4a186340f45c945fce1b17c85f98"}
Oct 30 21:26:21.471044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:21.470991    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz" podStartSLOduration=1.9407566790000002 podStartE2EDuration="2.470977245s" totalImagesPullingTime="530.220566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:21.468368665 +0000 UTC m=+1489.818096061" watchObservedRunningTime="2026-10-30 21:26:21.470977245 +0000 UTC m=+1489.820704641"
Oct 30 21:26:24.617696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:24.614793    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:24.617696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:24.614865    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.370719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:25.354782    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:25.370719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:25.354857    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.533712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:25.525728    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:25.533712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:25.525813    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.751183 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:26:25.751142    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:26:26.010758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:25.998850    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:26.010758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:25.998926    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:27.219899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:27.219857    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:27.237697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:27.230776    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:27.471519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:27.471432    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:26:28.216500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.216465    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"]
Oct 30 21:26:28.221291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.221266    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.231287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.231257    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-06bb0f-openshift-default-dockercfg-r6whw\""
Oct 30 21:26:28.240303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.240274    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"]
Oct 30 21:26:28.346599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.346549    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istiod-ca-cert\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.347075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.347042    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-credential-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.347283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.347252    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-data\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.347433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.347402    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-podinfo\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.347547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.347532    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjl5w\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-kube-api-access-qjl5w\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.347709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.347694    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.347900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.347883    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-envoy\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.348012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.347997    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-token\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.348201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.348184    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-certs\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.448700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.448302    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.450565    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-data\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.450615    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-podinfo\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.450645    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjl5w\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-kube-api-access-qjl5w\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.450700    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.450770    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-envoy\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.450796    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-token\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.450872    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-certs\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.450936    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istiod-ca-cert\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.451054    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-credential-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.451830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.451613    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-credential-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.457819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.452802    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-data\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.458377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.458352    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-certs\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.459491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.459451    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istiod-ca-cert\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.460330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.460218    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.481019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.468346    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:26:28.516786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.508423    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-podinfo\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.516786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.508522    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjl5w\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-kube-api-access-qjl5w\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.516786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.509147    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-token\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.531794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.530224    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-envoy\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.538845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.537905    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:28.859947 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:28.859914    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d132a9_f682_4d56_9de2_aaaa58c7eb35.slice/crio-68fccfef0d299348c9e7a3d4e1b3923284bc043c77b846f4f3c1d5281674508b WatchSource:0}: Error finding container 68fccfef0d299348c9e7a3d4e1b3923284bc043c77b846f4f3c1d5281674508b: Status 404 returned error can't find the container with id 68fccfef0d299348c9e7a3d4e1b3923284bc043c77b846f4f3c1d5281674508b
Oct 30 21:26:28.861777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.861749    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"]
Oct 30 21:26:28.897863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.897620    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:28.897863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.897737    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:28.897863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:28.897778    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:29.039424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.039374    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"]
Oct 30 21:26:29.046158 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.046127    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.073956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.070926    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"]
Oct 30 21:26:29.169765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.169726    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.169954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.169780    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.169954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.169846    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.169954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.169879    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.169954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.169920    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55qd\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-kube-api-access-l55qd\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.170200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.169979    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.170200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.170012    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.170200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.170034    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.170200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.170065    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.271605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.271555    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.271635    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.271741    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.271805    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.271842    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55qd\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-kube-api-access-l55qd\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.271931    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.271987    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.272016    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.272068    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.272086    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.272432    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.272513    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.272763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.272741    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.273113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.273085    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.275231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.275163    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.275489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.275465    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.285145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.285113    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.286439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.286410    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55qd\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-kube-api-access-l55qd\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.403011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.402974    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:29.483345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.482245    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" event={"ID":"62d132a9-f682-4d56-9de2-aaaa58c7eb35","Type":"ContainerStarted","Data":"b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1"}
Oct 30 21:26:29.483345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.482315    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" event={"ID":"62d132a9-f682-4d56-9de2-aaaa58c7eb35","Type":"ContainerStarted","Data":"68fccfef0d299348c9e7a3d4e1b3923284bc043c77b846f4f3c1d5281674508b"}
Oct 30 21:26:29.521738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.521656    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podStartSLOduration=1.521637428 podStartE2EDuration="1.521637428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:29.518798644 +0000 UTC m=+1497.868526041" watchObservedRunningTime="2026-10-30 21:26:29.521637428 +0000 UTC m=+1497.871364827"
Oct 30 21:26:29.538641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.538602    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:29.758852 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:29.758818    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108b52bf_e469_4ddf_80a7_5cace0eaf9fa.slice/crio-fd48f9b11eff74224516d2f01b6baca4e79dba4e52a9f41010d7807fd53ed851 WatchSource:0}: Error finding container fd48f9b11eff74224516d2f01b6baca4e79dba4e52a9f41010d7807fd53ed851: Status 404 returned error can't find the container with id fd48f9b11eff74224516d2f01b6baca4e79dba4e52a9f41010d7807fd53ed851
Oct 30 21:26:29.759528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.759505    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"]
Oct 30 21:26:29.762453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.762412    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:29.762570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.762504    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:29.762570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:29.762553    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:30.491429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:30.491376    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" event={"ID":"108b52bf-e469-4ddf-80a7-5cace0eaf9fa","Type":"ContainerStarted","Data":"eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e"}
Oct 30 21:26:30.491870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:30.491486    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" event={"ID":"108b52bf-e469-4ddf-80a7-5cace0eaf9fa","Type":"ContainerStarted","Data":"fd48f9b11eff74224516d2f01b6baca4e79dba4e52a9f41010d7807fd53ed851"}
Oct 30 21:26:30.534692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:30.533745    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podStartSLOduration=1.533724353 podStartE2EDuration="1.533724353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:30.528899726 +0000 UTC m=+1498.878627123" watchObservedRunningTime="2026-10-30 21:26:30.533724353 +0000 UTC m=+1498.883451750"
Oct 30 21:26:30.546655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:30.546621    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:31.403413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:31.403367    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:31.405219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:31.405191    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 21:26:31.405368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:31.405249    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 21:26:31.500168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:31.500138    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:31.501030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:31.501009    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:26:31.969122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:31.969085    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"]
Oct 30 21:26:31.978435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:31.978406    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"]
Oct 30 21:26:31.978704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:31.978547    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.004020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.003988    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-15e79c-openshift-default-dockercfg-vhqh5\""
Oct 30 21:26:32.142755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.142591    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:26:32.157147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.157117    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:26:32.167865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.167835    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:26:32.171064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171024    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-credential-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.171260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171102    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-data\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.171260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171160    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-istio-token\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.171260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171216    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.185978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171290    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-certs\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.185978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171327    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26abf544-f44b-43da-9cbf-50968294c4dc-istiod-ca-cert\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.185978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171406    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26abf544-f44b-43da-9cbf-50968294c4dc-istio-podinfo\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.185978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171458    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-envoy\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.185978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.171481    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsgb\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-kube-api-access-5hsgb\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.204701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.203892    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298197    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-credential-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298284    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-data\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298340    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-istio-token\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298393    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298499    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-certs\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298538    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26abf544-f44b-43da-9cbf-50968294c4dc-istiod-ca-cert\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298624    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26abf544-f44b-43da-9cbf-50968294c4dc-istio-podinfo\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298691    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-envoy\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.298751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.298719    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsgb\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-kube-api-access-5hsgb\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.324059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.299698    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-credential-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.324059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.299971    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-data\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.324059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.300462    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.324059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.300733    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-certs\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.324059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.301368    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26abf544-f44b-43da-9cbf-50968294c4dc-istiod-ca-cert\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.359266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.335470    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-envoy\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.359266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.336749    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26abf544-f44b-43da-9cbf-50968294c4dc-istio-podinfo\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.377697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.376552    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-istio-token\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.401696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.395900    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsgb\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-kube-api-access-5hsgb\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:32.411748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.403578    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 21:26:32.411748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.403637    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 21:26:32.658338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:32.658294    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:33.257550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.257518    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"]
Oct 30 21:26:33.262292 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:33.262260    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26abf544_f44b_43da_9cbf_50968294c4dc.slice/crio-155fe516bae7c4ac429ce8e095470b884325ed548c66542f82df1c18091df96c WatchSource:0}: Error finding container 155fe516bae7c4ac429ce8e095470b884325ed548c66542f82df1c18091df96c: Status 404 returned error can't find the container with id 155fe516bae7c4ac429ce8e095470b884325ed548c66542f82df1c18091df96c
Oct 30 21:26:33.264749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.264708    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:33.264845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.264783    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:33.264845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.264815    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:33.403737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.403701    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 21:26:33.403898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.403792    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 21:26:33.549725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.549601    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"]
Oct 30 21:26:33.554371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.554347    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:33.570878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.569866    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-06bb0f-dockercfg-5l9hg\""
Oct 30 21:26:33.570878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.569914    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-06bb0f\""
Oct 30 21:26:33.591774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.585551    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"]
Oct 30 21:26:33.594514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.593238    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" event={"ID":"26abf544-f44b-43da-9cbf-50968294c4dc","Type":"ContainerStarted","Data":"88f9b54d330fd5a5e679081790c7797637ae0347a7eb6ce6160994c2c410239d"}
Oct 30 21:26:33.594514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.593305    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" event={"ID":"26abf544-f44b-43da-9cbf-50968294c4dc","Type":"ContainerStarted","Data":"155fe516bae7c4ac429ce8e095470b884325ed548c66542f82df1c18091df96c"}
Oct 30 21:26:33.802618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.799273    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:33.850693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.849716    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b7ef66-3f64-4331-88cf-6825ab28d5dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b\" (UID: \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:33.850693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.850266    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv564\" (UniqueName: \"kubernetes.io/projected/57b7ef66-3f64-4331-88cf-6825ab28d5dc-kube-api-access-gv564\") pod \"payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b\" (UID: \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:33.958827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.955184    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv564\" (UniqueName: \"kubernetes.io/projected/57b7ef66-3f64-4331-88cf-6825ab28d5dc-kube-api-access-gv564\") pod \"payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b\" (UID: \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:33.958827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.955425    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b7ef66-3f64-4331-88cf-6825ab28d5dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b\" (UID: \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:33.967180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.959815    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b7ef66-3f64-4331-88cf-6825ab28d5dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b\" (UID: \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:33.973164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.973105    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podStartSLOduration=2.97308383 podStartE2EDuration="2.97308383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:33.966283667 +0000 UTC m=+1502.316011077" watchObservedRunningTime="2026-10-30 21:26:33.97308383 +0000 UTC m=+1502.322811226"
Oct 30 21:26:33.980296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:33.980215    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv564\" (UniqueName: \"kubernetes.io/projected/57b7ef66-3f64-4331-88cf-6825ab28d5dc-kube-api-access-gv564\") pod \"payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b\" (UID: \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:34.175976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:34.175938    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:34.404289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:34.404260    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 21:26:34.404453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:34.404311    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 21:26:34.565111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:34.565079    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"]
Oct 30 21:26:34.565847 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:34.565821    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b7ef66_3f64_4331_88cf_6825ab28d5dc.slice/crio-b8af609d043b96bbea1983c07c93a84f7f2e71847894e81b7ea85ba444c3e921 WatchSource:0}: Error finding container b8af609d043b96bbea1983c07c93a84f7f2e71847894e81b7ea85ba444c3e921: Status 404 returned error can't find the container with id b8af609d043b96bbea1983c07c93a84f7f2e71847894e81b7ea85ba444c3e921
Oct 30 21:26:34.611156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:34.611113    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b" event={"ID":"57b7ef66-3f64-4331-88cf-6825ab28d5dc","Type":"ContainerStarted","Data":"b8af609d043b96bbea1983c07c93a84f7f2e71847894e81b7ea85ba444c3e921"}
Oct 30 21:26:34.698049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:34.698015    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:35.404201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.404166    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 21:26:35.404588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.404216    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 21:26:35.582868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.582828    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"]
Oct 30 21:26:35.588528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.588501    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.596728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.594479    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-128d2f-openshift-default-dockercfg-wtbxb\""
Oct 30 21:26:35.608745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.608716    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"]
Oct 30 21:26:35.673416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.664040    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b" event={"ID":"57b7ef66-3f64-4331-88cf-6825ab28d5dc","Type":"ContainerStarted","Data":"e2321fa8ee454b8f94bcdb6e81a15716dc0e464074af62b1a4380fbcafd9ec0e"}
Oct 30 21:26:35.673416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.664848    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:35.673416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.664887    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:35.673416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.666534    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:26:35.753710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.746647    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-certs\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.753710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.746791    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-envoy\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.799701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.767801    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-token\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.799701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.767986    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1092b6c6-b82a-4c8d-bf08-af328c036781-istiod-ca-cert\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.799701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.768046    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-data\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.799701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.768086    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-credential-socket\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.799701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.768188    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-podinfo\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.799701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.768288    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4pwv\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-kube-api-access-v4pwv\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.799701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.768331    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-socket\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.907463    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-podinfo\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.907569    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4pwv\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-kube-api-access-v4pwv\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.907611    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-socket\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.907687    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-certs\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.907761    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-envoy\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.914403    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-socket\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.938377    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-token\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.938574    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1092b6c6-b82a-4c8d-bf08-af328c036781-istiod-ca-cert\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.938648    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-data\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.938767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.938729    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-credential-socket\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.993708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.939444    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-credential-socket\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.993708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.940217    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-certs\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.993708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.943919    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1092b6c6-b82a-4c8d-bf08-af328c036781-istiod-ca-cert\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:35.993708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:35.944606    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-data\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:36.046692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.021058    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b" podStartSLOduration=3.021036112 podStartE2EDuration="3.021036112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:35.694078297 +0000 UTC m=+1504.043805707" watchObservedRunningTime="2026-10-30 21:26:36.021036112 +0000 UTC m=+1504.370763510"
Oct 30 21:26:36.137858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.075318    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-podinfo\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:36.137858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.082964    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-envoy\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:36.137858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.115597    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4pwv\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-kube-api-access-v4pwv\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:36.161304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.161272    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-token\") pod \"e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") " pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:36.165963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.165916    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"]
Oct 30 21:26:36.177219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.177192    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"]
Oct 30 21:26:36.177352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.177303    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.210205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.210132    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:36.278577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.278536    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.278768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.278707    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.278768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.278744    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.278896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.278879    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.278955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.278911    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.279040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.279016    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8zb\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-kube-api-access-6z8zb\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.279084 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.279068    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.279169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.279152    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.279210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.279193    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381147    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381190    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381334    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381419    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381451    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381560    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381590    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381610    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381632    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8zb\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-kube-api-access-6z8zb\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381708    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.381928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.381847    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.382103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.382073    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.382599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.382397    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.383496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.383030    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.384824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.384788    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.385836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.385809    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.395901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.395870    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.396018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.395920    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8zb\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-kube-api-access-6z8zb\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.405547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.405507    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 21:26:36.405988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.405569    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 21:26:36.495201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.495118    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:36.543352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.543313    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"]
Oct 30 21:26:36.552476 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:36.552440    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1092b6c6_b82a_4c8d_bf08_af328c036781.slice/crio-47e960fe24edfa344b331eec4fa956cd98ac9d397b46aa14785a13efa009682c WatchSource:0}: Error finding container 47e960fe24edfa344b331eec4fa956cd98ac9d397b46aa14785a13efa009682c: Status 404 returned error can't find the container with id 47e960fe24edfa344b331eec4fa956cd98ac9d397b46aa14785a13efa009682c
Oct 30 21:26:36.556586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.556546    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:36.556777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.556636    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:36.556777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.556708    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:36.681070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.680302    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" event={"ID":"1092b6c6-b82a-4c8d-bf08-af328c036781","Type":"ContainerStarted","Data":"730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a"}
Oct 30 21:26:36.681070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.680364    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" event={"ID":"1092b6c6-b82a-4c8d-bf08-af328c036781","Type":"ContainerStarted","Data":"47e960fe24edfa344b331eec4fa956cd98ac9d397b46aa14785a13efa009682c"}
Oct 30 21:26:36.707715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.706197    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podStartSLOduration=1.706179255 podStartE2EDuration="1.706179255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:36.701713633 +0000 UTC m=+1505.051441034" watchObservedRunningTime="2026-10-30 21:26:36.706179255 +0000 UTC m=+1505.055906652"
Oct 30 21:26:36.838489 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:26:36.838448    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd024cbfe_5e99_4237_bece_8ddfba6e3d3a.slice/crio-28bf59232d6b3705d050e05413b8ee276ddcb258f2dda20d30a0cd141ff89a3e WatchSource:0}: Error finding container 28bf59232d6b3705d050e05413b8ee276ddcb258f2dda20d30a0cd141ff89a3e: Status 404 returned error can't find the container with id 28bf59232d6b3705d050e05413b8ee276ddcb258f2dda20d30a0cd141ff89a3e
Oct 30 21:26:36.865697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.863824    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:36.865697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.863950    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:36.865697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.863997    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:36.865697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:36.865425    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"]
Oct 30 21:26:37.225696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.222766    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:37.241940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.230424    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:37.404235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.404193    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 21:26:37.404399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.404265    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 21:26:37.690642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.690606    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" event={"ID":"d024cbfe-5e99-4237-bece-8ddfba6e3d3a","Type":"ContainerStarted","Data":"c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f"}
Oct 30 21:26:37.691110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.690682    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" event={"ID":"d024cbfe-5e99-4237-bece-8ddfba6e3d3a","Type":"ContainerStarted","Data":"28bf59232d6b3705d050e05413b8ee276ddcb258f2dda20d30a0cd141ff89a3e"}
Oct 30 21:26:37.691179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.691131    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:37.692133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.692113    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:26:37.754556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:37.754493    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podStartSLOduration=2.754479525 podStartE2EDuration="2.754479525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:37.727281076 +0000 UTC m=+1506.077008484" watchObservedRunningTime="2026-10-30 21:26:37.754479525 +0000 UTC m=+1506.104206921"
Oct 30 21:26:38.495566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:38.495526    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:38.497038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:38.497006    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 21:26:38.497182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:38.497067    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 21:26:38.857751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:38.857625    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:38.858318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:38.858299    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:38.859149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:38.859128    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:26:39.108248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.108168    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"]
Oct 30 21:26:39.131445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.112810    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:39.248694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.246998    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fe890f-059c-478e-a61a-8f8dc565b237-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl\" (UID: \"d4fe890f-059c-478e-a61a-8f8dc565b237\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:39.248694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.247305    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvp5g\" (UniqueName: \"kubernetes.io/projected/d4fe890f-059c-478e-a61a-8f8dc565b237-kube-api-access-bvp5g\") pod \"payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl\" (UID: \"d4fe890f-059c-478e-a61a-8f8dc565b237\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:39.430771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.367408    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvp5g\" (UniqueName: \"kubernetes.io/projected/d4fe890f-059c-478e-a61a-8f8dc565b237-kube-api-access-bvp5g\") pod \"payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl\" (UID: \"d4fe890f-059c-478e-a61a-8f8dc565b237\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:39.430771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.394472    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fe890f-059c-478e-a61a-8f8dc565b237-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl\" (UID: \"d4fe890f-059c-478e-a61a-8f8dc565b237\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:39.466906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.439816    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"]
Oct 30 21:26:39.532563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.512311    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 21:26:39.532563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.512391    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 21:26:39.532563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.513613    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-15e79c-dockercfg-t5mxl\""
Oct 30 21:26:39.532563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.513815    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-15e79c\""
Oct 30 21:26:39.532563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.528896    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fe890f-059c-478e-a61a-8f8dc565b237-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl\" (UID: \"d4fe890f-059c-478e-a61a-8f8dc565b237\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:39.555731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.553089    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvp5g\" (UniqueName: \"kubernetes.io/projected/d4fe890f-059c-478e-a61a-8f8dc565b237-kube-api-access-bvp5g\") pod \"payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl\" (UID: \"d4fe890f-059c-478e-a61a-8f8dc565b237\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:39.557501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.557473    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:39.569048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.569018    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"]
Oct 30 21:26:39.575942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.575906    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"]
Oct 30 21:26:39.576089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.576021    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.593895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.593322    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-06bb0f-serving-cert\""
Oct 30 21:26:39.723118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.721842    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.723118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.721903    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7mrb\" (UniqueName: \"kubernetes.io/projected/749359a0-c74c-4899-8505-89f60d2dc987-kube-api-access-f7mrb\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.723118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.722131    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.723118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.722234    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/749359a0-c74c-4899-8505-89f60d2dc987-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.870705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.869108    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.870705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.869350    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/749359a0-c74c-4899-8505-89f60d2dc987-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.870705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.869515    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.870705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.869550    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7mrb\" (UniqueName: \"kubernetes.io/projected/749359a0-c74c-4899-8505-89f60d2dc987-kube-api-access-f7mrb\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.903025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.897232    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.921741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.921318    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.921741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.921451    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7mrb\" (UniqueName: \"kubernetes.io/projected/749359a0-c74c-4899-8505-89f60d2dc987-kube-api-access-f7mrb\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.932818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.931902    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/749359a0-c74c-4899-8505-89f60d2dc987-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:39.962734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:39.962695    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:40.212138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.211488    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"]
Oct 30 21:26:40.408708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.404054    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"]
Oct 30 21:26:40.495968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.495894    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 21:26:40.495968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.495953    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 21:26:40.808996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.808900    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl" event={"ID":"d4fe890f-059c-478e-a61a-8f8dc565b237","Type":"ContainerStarted","Data":"edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c"}
Oct 30 21:26:40.808996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.808951    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl" event={"ID":"d4fe890f-059c-478e-a61a-8f8dc565b237","Type":"ContainerStarted","Data":"d167e3083cb1f33697378433b1612f1eef8faa7458621b0824595173b3d0b8ac"}
Oct 30 21:26:40.808996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.808981    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:40.810460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.810434    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn" event={"ID":"749359a0-c74c-4899-8505-89f60d2dc987","Type":"ContainerStarted","Data":"e08e201952d27f88495095966a8c8c10d57da949101355dd5433e7595ca18751"}
Oct 30 21:26:40.840766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:40.835469    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl" podStartSLOduration=1.835448994 podStartE2EDuration="1.835448994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:40.835103328 +0000 UTC m=+1509.184830725" watchObservedRunningTime="2026-10-30 21:26:40.835448994 +0000 UTC m=+1509.185176391"
Oct 30 21:26:41.496480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:41.496441    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 21:26:41.497056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:41.496502    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 21:26:41.935697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:41.934864    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn" event={"ID":"749359a0-c74c-4899-8505-89f60d2dc987","Type":"ContainerStarted","Data":"95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e"}
Oct 30 21:26:41.935697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:41.934967    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:42.495794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:42.495748    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 21:26:42.495988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:42.495813    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 21:26:43.496307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:43.496269    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 21:26:43.496773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:43.496334    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 21:26:44.499697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.497165    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 21:26:44.499697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.497247    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 21:26:44.882460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.882344    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn" podStartSLOduration=5.274275473 podStartE2EDuration="5.882323467s" totalImagesPullingTime="608.047994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:41.974019656 +0000 UTC m=+1510.323747053" watchObservedRunningTime="2026-10-30 21:26:44.882323467 +0000 UTC m=+1513.232050869"
Oct 30 21:26:44.892212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.887747    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"]
Oct 30 21:26:44.900906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.893831    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:44.913774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.910662    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-726a18-serving-cert\""
Oct 30 21:26:44.913774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.913135    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"]
Oct 30 21:26:44.948592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.946512    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d863f5-b05e-4922-9f70-015323992554-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:44.948592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.946597    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:44.948592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.946729    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwht5\" (UniqueName: \"kubernetes.io/projected/d5d863f5-b05e-4922-9f70-015323992554-kube-api-access-jwht5\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:44.948592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:44.946845    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.068738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.065199    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d863f5-b05e-4922-9f70-015323992554-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.068738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.065305    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.068738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.065345    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwht5\" (UniqueName: \"kubernetes.io/projected/d5d863f5-b05e-4922-9f70-015323992554-kube-api-access-jwht5\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.068738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.065465    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.068738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.067413    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d863f5-b05e-4922-9f70-015323992554-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.107695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.107294    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.133460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.133381    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwht5\" (UniqueName: \"kubernetes.io/projected/d5d863f5-b05e-4922-9f70-015323992554-kube-api-access-jwht5\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.135385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.135127    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:45.285690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:45.277965    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:46.148863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.145861    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"]
Oct 30 21:26:46.177339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.177304    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.179808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.179777    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"]
Oct 30 21:26:46.246443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.246413    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.246874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.246847    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.247119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.247097    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.247312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.247293    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt7kx\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-kube-api-access-qt7kx\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.247566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.247548    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.247802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.247782    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.248809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.248788    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.249013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.248994    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.249259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.249243    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.380967    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.381045    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt7kx\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-kube-api-access-qt7kx\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.381102    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.381156    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.381217    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.381257    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.381327    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.381469    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.381588    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.382432    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.382750    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.382809    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.383015    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.383741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.383278    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.390142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.390080    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.393797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.393734    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.404537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.404488    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.405486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.405415    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt7kx\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-kube-api-access-qt7kx\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.465719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.458390    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"]
Oct 30 21:26:46.545142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.544791    4633 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:46.545142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.544857    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:46.569502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.565961    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:46.687270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.686022    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:26:46.780794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.780757    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:46.782486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.782441    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:46.786544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.786498    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:26:46.926181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.926124    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:46.926356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.926272    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:46.926568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:46.926370    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:26:47.241276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:47.241228    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"]
Oct 30 21:26:47.398709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:47.395171    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72" event={"ID":"d5d863f5-b05e-4922-9f70-015323992554","Type":"ContainerStarted","Data":"6e602f1813f984e690620b45226e765ad914e51cf9fcfbc81afe4c690df8f6d2"}
Oct 30 21:26:47.399833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:47.399783    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" event={"ID":"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946","Type":"ContainerStarted","Data":"ef0c6721ecc9cc50f0b305f7d161669897e6520414af6895eb5274ee1e938675"}
Oct 30 21:26:47.956337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:47.956308    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:26:48.408343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:48.408305    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72" event={"ID":"d5d863f5-b05e-4922-9f70-015323992554","Type":"ContainerStarted","Data":"1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d"}
Oct 30 21:26:48.408808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:48.408367    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:48.410136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:48.410109    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" event={"ID":"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946","Type":"ContainerStarted","Data":"76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32"}
Oct 30 21:26:48.448887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:48.447489    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72" podStartSLOduration=4.035825117 podStartE2EDuration="4.447467624s" totalImagesPullingTime="411.642507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:48.430127671 +0000 UTC m=+1516.779855066" watchObservedRunningTime="2026-10-30 21:26:48.447467624 +0000 UTC m=+1516.797195020"
Oct 30 21:26:48.458974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:48.458913    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podStartSLOduration=2.458897673 podStartE2EDuration="2.458897673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:48.454043159 +0000 UTC m=+1516.803770556" watchObservedRunningTime="2026-10-30 21:26:48.458897673 +0000 UTC m=+1516.808625136"
Oct 30 21:26:48.568162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:48.568122    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:48.579864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:48.579827    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:26:48.580185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:48.580146    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 21:26:49.501028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.500990    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"]
Oct 30 21:26:49.538414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.538382    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.550898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.550808    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"]
Oct 30 21:26:49.552736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.552652    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-128d2f-serving-cert\""
Oct 30 21:26:49.567003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.566975    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:26:49.567166 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.567028    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 21:26:49.643947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.643094    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.644284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.644244    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvkf\" (UniqueName: \"kubernetes.io/projected/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-kube-api-access-zkvkf\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.644449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.644433    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.644616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.644601    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.749573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.748935    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.749573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.749035    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvkf\" (UniqueName: \"kubernetes.io/projected/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-kube-api-access-zkvkf\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.749573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.749069    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.749573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.749147    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.750169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.750144    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.753583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.753499    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.756708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.755245    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.760768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.760088    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvkf\" (UniqueName: \"kubernetes.io/projected/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-kube-api-access-zkvkf\") pod \"maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:49.867775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:49.867734    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:50.184657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:50.184623    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"]
Oct 30 21:26:50.481857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:50.481770    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp" event={"ID":"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0","Type":"ContainerStarted","Data":"0ba3ec7e1c564152cc501c3eeb361c0f584de5c45d46629160808c2891a14ec6"}
Oct 30 21:26:50.567048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:50.567011    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:26:50.567483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:50.567069    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 21:26:51.490758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:51.490718    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp" event={"ID":"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0","Type":"ContainerStarted","Data":"40ac13810f42e031762d8913e597be1fe37642496a17f43d6b691532ffdeef22"}
Oct 30 21:26:51.490960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:51.490826    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:26:51.567271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:51.567236    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:26:51.567685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:51.567338    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 21:26:51.939208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:51.939177    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:26:51.966717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:51.966631    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp" podStartSLOduration=2.46806905 podStartE2EDuration="2.96661586s" totalImagesPullingTime="498.54681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:51.532246265 +0000 UTC m=+1519.881973663" watchObservedRunningTime="2026-10-30 21:26:51.96661586 +0000 UTC m=+1520.316343255"
Oct 30 21:26:52.573695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:52.571267    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:26:52.573695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:52.571342    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 21:26:53.567012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:53.566976    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:26:53.567198 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:53.567055    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 21:26:54.447277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:54.447251    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:26:54.566621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:54.566590    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 30 21:26:54.566814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:54.566645    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 30 21:26:55.016241 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:26:55.016200    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:26:55.795471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:55.795441    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:55.797490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:55.797470    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:55.798519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:55.798469    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:26:57.506927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:26:57.506892    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:27:05.845982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.845948    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"]
Oct 30 21:27:05.854010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.853988    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.866904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.866881    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"]
Oct 30 21:27:05.867656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.867486    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-d4a0a5-openshift-default-dockercfg-8sgkb\""
Oct 30 21:27:05.974025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.973987    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqvf\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-kube-api-access-frqvf\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.974218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.974045    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-token\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.974218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.974199    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-podinfo\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.974342 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.974269    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-data\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.974407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.974353    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-credential-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.974475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.974459    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-envoy\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.974518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.974484    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.974570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.974518    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-certs\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:05.974570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:05.974535    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istiod-ca-cert\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.075969    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-envoy\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076004    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076047    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-certs\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076067    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istiod-ca-cert\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076142    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqvf\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-kube-api-access-frqvf\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076165    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-token\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076306    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-podinfo\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076403    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-data\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076440    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-credential-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076509    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076554    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-certs\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.076856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076834    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-credential-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.077007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.076987    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-data\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.077007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.077001    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istiod-ca-cert\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.079010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.078975    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-envoy\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.079185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.079167    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-podinfo\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.153738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.143033    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqvf\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-kube-api-access-frqvf\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.153738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.147611    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-token\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.202692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.201461    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:06.452179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.450481    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"]
Oct 30 21:27:06.462662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.461526    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.467796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.467025    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-b30a42\"/\"openshift-service-ca.crt\""
Oct 30 21:27:06.467796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.467135    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-b30a42\"/\"default-dockercfg-82spm\""
Oct 30 21:27:06.469360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.469240    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-b30a42\"/\"auth-test-model-b30a42-kserve-self-signed-certs\""
Oct 30 21:27:06.469360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.469298    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-b30a42\"/\"kube-root-ca.crt\""
Oct 30 21:27:06.498898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.498865    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"]
Oct 30 21:27:06.615145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.615111    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-dshm\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.615316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.615287    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf7xh\" (UniqueName: \"kubernetes.io/projected/1271668c-67d5-410e-ae88-b44237083f30-kube-api-access-vf7xh\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.615407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.615390    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-tmp-dir\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.615490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.615477    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-model-cache\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.615524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.615500    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1271668c-67d5-410e-ae88-b44237083f30-tls-certs\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.615635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.615621    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-home\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.701087 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:27:06.701048    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff05b49_76c1_45dc_9f4d_3d07699c12d7.slice/crio-d19d91277c40d03e40a27499f67e0d9fc075a2f302cf443fa6cc7949e182a0d7 WatchSource:0}: Error finding container d19d91277c40d03e40a27499f67e0d9fc075a2f302cf443fa6cc7949e182a0d7: Status 404 returned error can't find the container with id d19d91277c40d03e40a27499f67e0d9fc075a2f302cf443fa6cc7949e182a0d7
Oct 30 21:27:06.701765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.701743    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"]
Oct 30 21:27:06.704271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.704237    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:06.704345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.704305    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:06.704345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.704336    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:06.717686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.717639    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-dshm\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.718079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.718055    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf7xh\" (UniqueName: \"kubernetes.io/projected/1271668c-67d5-410e-ae88-b44237083f30-kube-api-access-vf7xh\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.718311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.718287    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-tmp-dir\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.719019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.718998    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-model-cache\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.720641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.720620    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1271668c-67d5-410e-ae88-b44237083f30-tls-certs\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.720778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.719484    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-tmp-dir\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.720778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.719454    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-model-cache\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.720778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.720755    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-home\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.721063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.721042    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-home\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.722100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.722080    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-dshm\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.723430 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.723411    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1271668c-67d5-410e-ae88-b44237083f30-tls-certs\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.736608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.736589    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf7xh\" (UniqueName: \"kubernetes.io/projected/1271668c-67d5-410e-ae88-b44237083f30-kube-api-access-vf7xh\") pod \"auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") " pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:06.779662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:06.779637    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:07.289765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:07.289623    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"]
Oct 30 21:27:07.290174 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:27:07.289789    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1271668c_67d5_410e_ae88_b44237083f30.slice/crio-f0a680fc180c1e311722bc620a4403fd3f474a2955ff6d72a81f8dc1e9daf7b9 WatchSource:0}: Error finding container f0a680fc180c1e311722bc620a4403fd3f474a2955ff6d72a81f8dc1e9daf7b9: Status 404 returned error can't find the container with id f0a680fc180c1e311722bc620a4403fd3f474a2955ff6d72a81f8dc1e9daf7b9
Oct 30 21:27:07.671338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:07.671299    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" event={"ID":"3ff05b49-76c1-45dc-9f4d-3d07699c12d7","Type":"ContainerStarted","Data":"ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901"}
Oct 30 21:27:07.671553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:07.671364    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" event={"ID":"3ff05b49-76c1-45dc-9f4d-3d07699c12d7","Type":"ContainerStarted","Data":"d19d91277c40d03e40a27499f67e0d9fc075a2f302cf443fa6cc7949e182a0d7"}
Oct 30 21:27:07.673137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:07.673110    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf" event={"ID":"1271668c-67d5-410e-ae88-b44237083f30","Type":"ContainerStarted","Data":"91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c"}
Oct 30 21:27:07.673256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:07.673144    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf" event={"ID":"1271668c-67d5-410e-ae88-b44237083f30","Type":"ContainerStarted","Data":"f0a680fc180c1e311722bc620a4403fd3f474a2955ff6d72a81f8dc1e9daf7b9"}
Oct 30 21:27:07.710358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:07.710310    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podStartSLOduration=2.710294213 podStartE2EDuration="2.710294213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:07.706861108 +0000 UTC m=+1536.056588504" watchObservedRunningTime="2026-10-30 21:27:07.710294213 +0000 UTC m=+1536.060021609"
Oct 30 21:27:07.732944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:07.732890    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf" podStartSLOduration=1.732873 podStartE2EDuration="1.732873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:07.731230429 +0000 UTC m=+1536.080957826" watchObservedRunningTime="2026-10-30 21:27:07.732873 +0000 UTC m=+1536.082600397"
Oct 30 21:27:07.779804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:07.779777    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:08.142275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:08.142190    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:27:08.202900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:08.202853    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:08.212148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:08.211835    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:08.692703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:08.687688    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:08.692703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:08.689024    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:27:13.999967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:13.999792    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:16.779922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:16.779884    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:17.205325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:17.205197    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:17.508575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:17.508496    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:27:21.705880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.705819    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:27:21.706416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.705908    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:21.781071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.781018    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"]
Oct 30 21:27:21.881239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.881197    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:21.885709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.885643    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-904993-serving-cert\""
Oct 30 21:27:21.894752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.894722    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"]
Oct 30 21:27:21.913480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.913440    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktjl\" (UniqueName: \"kubernetes.io/projected/af90c02b-8a15-411b-bbcb-d3d658effbec-kube-api-access-9ktjl\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:21.913838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.913811    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af90c02b-8a15-411b-bbcb-d3d658effbec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:21.913954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.913922    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:21.914031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:21.914013    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.037750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.033356    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktjl\" (UniqueName: \"kubernetes.io/projected/af90c02b-8a15-411b-bbcb-d3d658effbec-kube-api-access-9ktjl\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.037750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.033630    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af90c02b-8a15-411b-bbcb-d3d658effbec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.037750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.033750    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.037750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.033896    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.050596 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.050565    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af90c02b-8a15-411b-bbcb-d3d658effbec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.052603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.052545    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.055831 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.055769    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.059574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.059544    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktjl\" (UniqueName: \"kubernetes.io/projected/af90c02b-8a15-411b-bbcb-d3d658effbec-kube-api-access-9ktjl\") pod \"maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.258920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.258877    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:22.822025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:22.821999    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"]
Oct 30 21:27:22.823059 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:27:22.823034    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf90c02b_8a15_411b_bbcb_d3d658effbec.slice/crio-19a44763694418dfee0e7ac8e91d14f0c7ec8eb66b35f79c8666655b1d1e6eaf WatchSource:0}: Error finding container 19a44763694418dfee0e7ac8e91d14f0c7ec8eb66b35f79c8666655b1d1e6eaf: Status 404 returned error can't find the container with id 19a44763694418dfee0e7ac8e91d14f0c7ec8eb66b35f79c8666655b1d1e6eaf
Oct 30 21:27:23.629382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:23.627922    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp" event={"ID":"af90c02b-8a15-411b-bbcb-d3d658effbec","Type":"ContainerStarted","Data":"19a44763694418dfee0e7ac8e91d14f0c7ec8eb66b35f79c8666655b1d1e6eaf"}
Oct 30 21:27:23.732860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:23.732826    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:27:23.733030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:23.732888    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:24.665902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:24.641577    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp" event={"ID":"af90c02b-8a15-411b-bbcb-d3d658effbec","Type":"ContainerStarted","Data":"8c810d75fc8299393768de5c4376e53b54a31587111c352f80914354d3a1b08b"}
Oct 30 21:27:24.665902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:24.641658    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:24.665902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:24.665828    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp" podStartSLOduration=2.465997344 podStartE2EDuration="3.665806951s" totalImagesPullingTime="1.199809607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:24.663175825 +0000 UTC m=+1553.012903221" watchObservedRunningTime="2026-10-30 21:27:24.665806951 +0000 UTC m=+1553.015534348"
Oct 30 21:27:25.019851 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:27:25.019816    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:27:25.457280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.457246    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"]
Oct 30 21:27:25.462892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.462864    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.472465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.472394    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-d4a0a5-serving-cert\""
Oct 30 21:27:25.479530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.479505    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"]
Oct 30 21:27:25.573790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.573763    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.573952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.573929    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ac7f8-6268-40e8-9599-5c11cca9390d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.574025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.573971    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpzm\" (UniqueName: \"kubernetes.io/projected/0a3ac7f8-6268-40e8-9599-5c11cca9390d-kube-api-access-nvpzm\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.574260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.574234    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.651608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.651572    4633 generic.go:356] "Generic (PLEG): container finished" podID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerID="730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a" exitCode=0
Oct 30 21:27:25.651857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.651647    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" event={"ID":"1092b6c6-b82a-4c8d-bf08-af328c036781","Type":"ContainerDied","Data":"730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a"}
Oct 30 21:27:25.652238 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.652220    4633 scope.go:122] "RemoveContainer" containerID="730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a"
Oct 30 21:27:25.653562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.653542    4633 generic.go:356] "Generic (PLEG): container finished" podID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerID="b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1" exitCode=0
Oct 30 21:27:25.653747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.653620    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" event={"ID":"62d132a9-f682-4d56-9de2-aaaa58c7eb35","Type":"ContainerDied","Data":"b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1"}
Oct 30 21:27:25.654099 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.654081    4633 scope.go:122] "RemoveContainer" containerID="b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1"
Oct 30 21:27:25.654297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.654220    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.654372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.654319    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.654437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.654371    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.655780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.655742    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.655881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.655817    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.655881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.655853    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.658461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.656917    4633 generic.go:356] "Generic (PLEG): container finished" podID="26abf544-f44b-43da-9cbf-50968294c4dc" containerID="88f9b54d330fd5a5e679081790c7797637ae0347a7eb6ce6160994c2c410239d" exitCode=0
Oct 30 21:27:25.658461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.657634    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" event={"ID":"26abf544-f44b-43da-9cbf-50968294c4dc","Type":"ContainerDied","Data":"88f9b54d330fd5a5e679081790c7797637ae0347a7eb6ce6160994c2c410239d"}
Oct 30 21:27:25.658461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.658044    4633 scope.go:122] "RemoveContainer" containerID="88f9b54d330fd5a5e679081790c7797637ae0347a7eb6ce6160994c2c410239d"
Oct 30 21:27:25.688966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.677517    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.688966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.677704    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.688966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.677844    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ac7f8-6268-40e8-9599-5c11cca9390d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.688966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.677875    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpzm\" (UniqueName: \"kubernetes.io/projected/0a3ac7f8-6268-40e8-9599-5c11cca9390d-kube-api-access-nvpzm\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.688966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.678923    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.688966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.679037    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.688966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.679091    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:25.688966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.684596    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ac7f8-6268-40e8-9599-5c11cca9390d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.723252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.723215    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.725204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.725148    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpzm\" (UniqueName: \"kubernetes.io/projected/0a3ac7f8-6268-40e8-9599-5c11cca9390d-kube-api-access-nvpzm\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.762817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.726348    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:25.793697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:25.793640    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:26.310947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:26.310512    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"]
Oct 30 21:27:26.320621 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:27:26.320579    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3ac7f8_6268_40e8_9599_5c11cca9390d.slice/crio-69cf4364e68eebac29aa329f42ed769b16baf379a972bee7c54df1818eda0d99 WatchSource:0}: Error finding container 69cf4364e68eebac29aa329f42ed769b16baf379a972bee7c54df1818eda0d99: Status 404 returned error can't find the container with id 69cf4364e68eebac29aa329f42ed769b16baf379a972bee7c54df1818eda0d99
Oct 30 21:27:26.686733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:26.686633    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" event={"ID":"1092b6c6-b82a-4c8d-bf08-af328c036781","Type":"ContainerStarted","Data":"42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886"}
Oct 30 21:27:26.689859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:26.689826    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" event={"ID":"62d132a9-f682-4d56-9de2-aaaa58c7eb35","Type":"ContainerStarted","Data":"87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968"}
Oct 30 21:27:26.691086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:26.691057    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p" event={"ID":"0a3ac7f8-6268-40e8-9599-5c11cca9390d","Type":"ContainerStarted","Data":"69cf4364e68eebac29aa329f42ed769b16baf379a972bee7c54df1818eda0d99"}
Oct 30 21:27:26.693182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:26.693162    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" event={"ID":"26abf544-f44b-43da-9cbf-50968294c4dc","Type":"ContainerStarted","Data":"c6ed724bd4f6ea3abec644aa8056d034487cc465b01f3c010d9e43e502bb8544"}
Oct 30 21:27:27.211697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.211642    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:27:27.212178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.212152    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:27:27.212385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.212357    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 21:27:27.412705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.411734    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"]
Oct 30 21:27:27.438703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.436964    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.480754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.459055    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-128d2f\"/\"auth-test-model-128d2f-kserve-self-signed-certs\""
Oct 30 21:27:27.481009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.465982    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-128d2f\"/\"default-dockercfg-rzd5t\""
Oct 30 21:27:27.481108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.467709    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-128d2f\"/\"kube-root-ca.crt\""
Oct 30 21:27:27.481188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.467780    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-128d2f\"/\"openshift-service-ca.crt\""
Oct 30 21:27:27.505935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.503298    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-dshm\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.505935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.503355    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrrz\" (UniqueName: \"kubernetes.io/projected/d648c9dd-5abf-460a-9d76-af50203d5df3-kube-api-access-psrrz\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.505935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.503457    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-model-cache\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.505935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.503509    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-tmp-dir\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.505935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.503643    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-home\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.505935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.503723    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d648c9dd-5abf-460a-9d76-af50203d5df3-tls-certs\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.570695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.548749    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.573822    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.573887    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.596744    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"]
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.636950    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-model-cache\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.637015    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-tmp-dir\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.637141    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-home\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.637165    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d648c9dd-5abf-460a-9d76-af50203d5df3-tls-certs\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.637385    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-dshm\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.637418    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrrz\" (UniqueName: \"kubernetes.io/projected/d648c9dd-5abf-460a-9d76-af50203d5df3-kube-api-access-psrrz\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.638378    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-model-cache\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.666707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.638632    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-tmp-dir\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.689856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.689816    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:27:27.709818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.702197    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-home\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:27.775621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.775528    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:27:27.883694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:27.874505    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:27:28.016696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.000516    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d648c9dd-5abf-460a-9d76-af50203d5df3-tls-certs\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:28.017575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.012058    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-dshm\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:28.062993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.062863    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p" event={"ID":"0a3ac7f8-6268-40e8-9599-5c11cca9390d","Type":"ContainerStarted","Data":"33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959"}
Oct 30 21:27:28.077869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.063596    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrrz\" (UniqueName: \"kubernetes.io/projected/d648c9dd-5abf-460a-9d76-af50203d5df3-kube-api-access-psrrz\") pod \"auth-test-model-128d2f-kserve-868cc88fc7-qkw6t\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") " pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:28.078228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.078200    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:28.159860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.155178    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:28.220326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.219858    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:27:28.220326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.219922    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 21:27:28.539692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.539119    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 21:27:28.539692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.539187    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 21:27:28.674695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.670091    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:27:28.674695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:28.670157    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:27:29.224697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:29.219143    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:27:29.247699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:29.244910    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 21:27:29.561761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:29.559904    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 21:27:29.561761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:29.559971    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 21:27:29.666699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:29.653363    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p" podStartSLOduration=4.233162153 podStartE2EDuration="4.65333956s" totalImagesPullingTime="420.177407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:28.107319774 +0000 UTC m=+1556.457047194" watchObservedRunningTime="2026-10-30 21:27:29.65333956 +0000 UTC m=+1558.003066956"
Oct 30 21:27:29.670694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:29.667742    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"]
Oct 30 21:27:29.700736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:29.695800    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:27:29.700736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:29.695872    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:27:29.700736 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:27:29.696864    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd648c9dd_5abf_460a_9d76_af50203d5df3.slice/crio-0967acb857a12a91cf1dff5fa92a5646c6e74b44e459c0fd62b51075a0ec5151 WatchSource:0}: Error finding container 0967acb857a12a91cf1dff5fa92a5646c6e74b44e459c0fd62b51075a0ec5151: Status 404 returned error can't find the container with id 0967acb857a12a91cf1dff5fa92a5646c6e74b44e459c0fd62b51075a0ec5151
Oct 30 21:27:30.219828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.214031    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:27:30.219828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.214094    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 21:27:30.554790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.554415    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 21:27:30.554790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.554511    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 21:27:30.560570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.560529    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t" event={"ID":"d648c9dd-5abf-460a-9d76-af50203d5df3","Type":"ContainerStarted","Data":"a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247"}
Oct 30 21:27:30.560855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.560839    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t" event={"ID":"d648c9dd-5abf-460a-9d76-af50203d5df3","Type":"ContainerStarted","Data":"0967acb857a12a91cf1dff5fa92a5646c6e74b44e459c0fd62b51075a0ec5151"}
Oct 30 21:27:30.679925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.679886    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:27:30.701147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.701099    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:27:30.710654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.710614    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:27:30.835776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:30.813907    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t" podStartSLOduration=3.813885449 podStartE2EDuration="3.813885449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:30.619063361 +0000 UTC m=+1558.968790763" watchObservedRunningTime="2026-10-30 21:27:30.813885449 +0000 UTC m=+1559.163612845"
Oct 30 21:27:31.193313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:31.193282    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:31.210532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:31.210499    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:27:31.210784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:31.210758    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 21:27:31.507133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:31.507038    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:27:31.551046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:31.551008    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 21:27:31.551341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:31.551313    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 21:27:31.670696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:31.667078    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:27:31.691693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:31.680895    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:27:32.211556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:32.211520    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:27:32.211774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:32.211606    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 21:27:32.542761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:32.542149    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 21:27:32.542761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:32.542217    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 21:27:32.686696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:32.676492    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:27:32.686696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:32.676557    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:27:33.210542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:33.210502    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:27:33.210754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:33.210580    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 21:27:33.548758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:33.544084    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 21:27:33.548758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:33.544147    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 21:27:33.673711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:33.669179    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:27:33.673711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:33.669243    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:27:34.225693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:34.224977    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 30 21:27:34.225693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:34.225040    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 30 21:27:34.392695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:34.391956    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:27:34.543759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:34.541078    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 30 21:27:34.543759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:34.541138    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 30 21:27:34.690931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:34.684873    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:27:34.690931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:34.684949    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 21:27:35.469452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:35.469410    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:27:35.470576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:35.470532    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:27:35.471599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:35.471576    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:27:35.557950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:35.557920    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:27:35.560114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:35.560089    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:27:35.664887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:35.664794    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:27:35.676648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:35.676081    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:27:35.712936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:35.712902    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:27:38.155864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:38.155829    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:38.171394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:38.171362    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:38.975995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:38.975961    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:27:44.414949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:44.414831    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:27:44.414949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:44.414896    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:45.901224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:45.901188    4633 generic.go:356] "Generic (PLEG): container finished" podID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerID="939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0" exitCode=0
Oct 30 21:27:45.901651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:45.901264    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" event={"ID":"c175ae88-19d0-4c92-8f35-649ac11138ac","Type":"ContainerDied","Data":"939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0"}
Oct 30 21:27:45.901761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:45.901746    4633 scope.go:122] "RemoveContainer" containerID="939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0"
Oct 30 21:27:45.903518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:45.903488    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:45.903597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:45.903566    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:45.903650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:45.903600    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:46.912309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:46.912272    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" event={"ID":"c175ae88-19d0-4c92-8f35-649ac11138ac","Type":"ContainerStarted","Data":"ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05"}
Oct 30 21:27:47.144765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:47.144718    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:27:47.145256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:47.145234    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 21:27:47.145319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:47.145289    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 21:27:47.844966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:47.844928    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"]
Oct 30 21:27:47.849920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:47.849898    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:47.875007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:47.874978    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-a9230f-openshift-default-dockercfg-7qqkw\""
Oct 30 21:27:47.880640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:47.880612    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"]
Oct 30 21:27:48.001030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.000998    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.001566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.001084    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.001566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.001146    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hxpt\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-kube-api-access-4hxpt\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.001566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.001182    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.001566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.001252    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68456ea2-bbf0-467a-8c03-b371d72ac3be-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.001566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.001300    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.001566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.001339    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.001566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.001409    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.001566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.001431    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.102888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.102793    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.102888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.102868    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103042    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hxpt\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-kube-api-access-4hxpt\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103094    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103223    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68456ea2-bbf0-467a-8c03-b371d72ac3be-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103302    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103333    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103311    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103436    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103570    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103605    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103847    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.103973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.103953    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68456ea2-bbf0-467a-8c03-b371d72ac3be-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.104202 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.104032    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.105888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.105864    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.107109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.107083    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.140552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.140518    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.140866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.140752    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hxpt\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-kube-api-access-4hxpt\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.145719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.145636    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 21:27:48.145719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.145716    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 21:27:48.164327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.164295    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:48.544722 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:27:48.544000    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68456ea2_bbf0_467a_8c03_b371d72ac3be.slice/crio-f143d6b232d4e10cdcf13ed3895da102b9447f9fe022bf94b028e81ab4ef2bd9 WatchSource:0}: Error finding container f143d6b232d4e10cdcf13ed3895da102b9447f9fe022bf94b028e81ab4ef2bd9: Status 404 returned error can't find the container with id f143d6b232d4e10cdcf13ed3895da102b9447f9fe022bf94b028e81ab4ef2bd9
Oct 30 21:27:48.544722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.544200    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"]
Oct 30 21:27:48.547107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.547073    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:48.547207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.547138    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:48.547207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.547169    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:27:48.988820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.988778    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" event={"ID":"68456ea2-bbf0-467a-8c03-b371d72ac3be","Type":"ContainerStarted","Data":"6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004"}
Oct 30 21:27:48.989189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:48.989166    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" event={"ID":"68456ea2-bbf0-467a-8c03-b371d72ac3be","Type":"ContainerStarted","Data":"f143d6b232d4e10cdcf13ed3895da102b9447f9fe022bf94b028e81ab4ef2bd9"}
Oct 30 21:27:49.082747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:49.082652    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podStartSLOduration=2.082631368 podStartE2EDuration="2.082631368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:49.053637721 +0000 UTC m=+1577.403365118" watchObservedRunningTime="2026-10-30 21:27:49.082631368 +0000 UTC m=+1577.432358767"
Oct 30 21:27:49.145019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:49.144981    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 21:27:49.145229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:49.145041    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 21:27:49.165465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:49.165435    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:49.167048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:49.167024    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 21:27:49.167160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:49.167071    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 21:27:50.148545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:50.148098    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 21:27:50.148545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:50.148159    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 21:27:50.166645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:50.166526    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 21:27:50.166645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:50.166585    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 21:27:51.146696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:51.146127    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 21:27:51.146696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:51.146193    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 21:27:51.171697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:51.171059    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 21:27:51.171697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:51.171136    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 21:27:52.153696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:52.145118    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 21:27:52.153696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:52.145199    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 21:27:52.176264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:52.175918    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 21:27:52.176264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:52.175982    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 21:27:53.145113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:53.145075    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 21:27:53.145319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:53.145142    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 21:27:53.189707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:53.180449    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 21:27:53.190357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:53.190322    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 21:27:54.149461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:54.149429    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:27:54.149911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:54.149890    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:27:54.150758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:54.150734    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:27:54.165560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:54.165525    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 21:27:54.165773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:54.165597    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 21:27:55.043746 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:27:55.043710    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:27:55.177695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:55.168143    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 21:27:55.177695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:55.168214    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 21:27:56.171095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:56.171063    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:56.171820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:56.171793    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:56.172899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:56.172878    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:27:58.582499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.582464    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"]
Oct 30 21:27:58.588524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.588502    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.607823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.607794    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-15e79c\"/\"openshift-service-ca.crt\""
Oct 30 21:27:58.608022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.607893    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-15e79c\"/\"default-dockercfg-8pwnl\""
Oct 30 21:27:58.608022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.607910    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-15e79c\"/\"kube-root-ca.crt\""
Oct 30 21:27:58.608844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.608823    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-15e79c\"/\"auth-test-model-15e79c-kserve-self-signed-certs\""
Oct 30 21:27:58.611332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.611306    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"]
Oct 30 21:27:58.637840 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.637808    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-dshm\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.638029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.637963    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-model-cache\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.638120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.638028    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3004e018-fc29-48a1-b7a8-a71fb646cf63-tls-certs\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.638185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.638117    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g92v\" (UniqueName: \"kubernetes.io/projected/3004e018-fc29-48a1-b7a8-a71fb646cf63-kube-api-access-5g92v\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.638348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.638167    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-tmp-dir\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.639719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.638663    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-home\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.740573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.740536    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-dshm\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.740811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.740655    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-model-cache\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.740811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.740773    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3004e018-fc29-48a1-b7a8-a71fb646cf63-tls-certs\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.740936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.740817    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g92v\" (UniqueName: \"kubernetes.io/projected/3004e018-fc29-48a1-b7a8-a71fb646cf63-kube-api-access-5g92v\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.740936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.740868    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-tmp-dir\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.740936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.740892    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-home\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.741138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.741069    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-model-cache\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.741259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.741231    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-tmp-dir\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.741369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.741345    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-home\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.743147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.743124    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-dshm\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.743367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.743348    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3004e018-fc29-48a1-b7a8-a71fb646cf63-tls-certs\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.757943 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.757917    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g92v\" (UniqueName: \"kubernetes.io/projected/3004e018-fc29-48a1-b7a8-a71fb646cf63-kube-api-access-5g92v\") pod \"auth-test-model-15e79c-kserve-598d599d77-bqfck\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") " pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:58.905693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:58.904008    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:27:59.342963 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:27:59.342937    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3004e018_fc29_48a1_b7a8_a71fb646cf63.slice/crio-a51c6c76ed791f8ab893fe05f15f958573527ebdb445d9e28d086cc6c45235be WatchSource:0}: Error finding container a51c6c76ed791f8ab893fe05f15f958573527ebdb445d9e28d086cc6c45235be: Status 404 returned error can't find the container with id a51c6c76ed791f8ab893fe05f15f958573527ebdb445d9e28d086cc6c45235be
Oct 30 21:27:59.343160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:27:59.343129    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"]
Oct 30 21:28:00.199056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:00.199014    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck" event={"ID":"3004e018-fc29-48a1-b7a8-a71fb646cf63","Type":"ContainerStarted","Data":"3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037"}
Oct 30 21:28:00.199592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:00.199065    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck" event={"ID":"3004e018-fc29-48a1-b7a8-a71fb646cf63","Type":"ContainerStarted","Data":"a51c6c76ed791f8ab893fe05f15f958573527ebdb445d9e28d086cc6c45235be"}
Oct 30 21:28:00.257394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:00.257336    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck" podStartSLOduration=2.25732122 podStartE2EDuration="2.25732122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:00.251349206 +0000 UTC m=+1588.601076602" watchObservedRunningTime="2026-10-30 21:28:00.25732122 +0000 UTC m=+1588.607048644"
Oct 30 21:28:00.907702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:00.906802    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:28:03.227955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:03.227911    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:28:08.904716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:08.904635    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:28:08.919211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:08.919176    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:28:09.394973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:09.394871    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:28:11.526350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.526316    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"]
Oct 30 21:28:11.532202 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.532180    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.536974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.536950    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ed261\"/\"openshift-service-ca.crt\""
Oct 30 21:28:11.537855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.537837    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ed261\"/\"kube-root-ca.crt\""
Oct 30 21:28:11.537953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.537895    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ed261\"/\"default-dockercfg-swfcw\""
Oct 30 21:28:11.538070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.538056    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6ed261\"/\"auth-test-model-6ed261-kserve-self-signed-certs\""
Oct 30 21:28:11.552486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.552465    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"]
Oct 30 21:28:11.691152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.691115    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-model-cache\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.691339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.691283    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b44g\" (UniqueName: \"kubernetes.io/projected/a41b8717-7029-4354-ae73-a6df0b7d6b7b-kube-api-access-6b44g\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.691398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.691380    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tls-certs\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.691439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.691409    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tmp-dir\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.691560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.691539    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-home\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.691602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.691592    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-dshm\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.792491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.792404    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b44g\" (UniqueName: \"kubernetes.io/projected/a41b8717-7029-4354-ae73-a6df0b7d6b7b-kube-api-access-6b44g\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.792491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.792486    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tls-certs\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.792936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.792517    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tmp-dir\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.792936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.792601    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-home\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.792936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.792757    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-dshm\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.792936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.792875    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-model-cache\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.793155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.793130    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-home\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.794439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.794372    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-model-cache\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.796308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.793146    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tmp-dir\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.806281 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.806244    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-dshm\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.806502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.806479    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tls-certs\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:11.891983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:11.891914    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b44g\" (UniqueName: \"kubernetes.io/projected/a41b8717-7029-4354-ae73-a6df0b7d6b7b-kube-api-access-6b44g\") pod \"auth-test-model-6ed261-kserve-58dd98d645-wwd2d\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") " pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:12.146958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:12.146873    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:12.507650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:12.507622    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"]
Oct 30 21:28:13.419807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:13.419768    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d" event={"ID":"a41b8717-7029-4354-ae73-a6df0b7d6b7b","Type":"ContainerStarted","Data":"e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314"}
Oct 30 21:28:13.420212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:13.419821    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d" event={"ID":"a41b8717-7029-4354-ae73-a6df0b7d6b7b","Type":"ContainerStarted","Data":"224fa435abdf22551935d69914c93e65887959ce65977d1701bc61d54703064e"}
Oct 30 21:28:14.147931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:14.147904    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:17.977270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:17.977210    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d" podStartSLOduration=6.977194228 podStartE2EDuration="6.977194228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:13.495870043 +0000 UTC m=+1601.845597438" watchObservedRunningTime="2026-10-30 21:28:17.977194228 +0000 UTC m=+1606.326921624"
Oct 30 21:28:17.980080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:17.980051    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"]
Oct 30 21:28:17.984964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:17.984941    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.024503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.024471    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"]
Oct 30 21:28:18.105848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.105820    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"]
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.114548    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.114887    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.114964    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.115080    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhntf\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-kube-api-access-dhntf\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.115106    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.115205    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.115244    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.115279    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.115437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.115366    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.116076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.115615    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.130899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.130873    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"]
Oct 30 21:28:18.216860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.216819    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.217065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.216890    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhntf\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-kube-api-access-dhntf\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.216959    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217001    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.217193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217104    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217195    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217238    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217282    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2kr\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-kube-api-access-dt2kr\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.217335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217317    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c630ce-a087-471a-8916-1d97307638fd-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.217448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217384    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217405    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217446    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217452    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.217575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217510    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c630ce-a087-471a-8916-1d97307638fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.217826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217588    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.217826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217664    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217738    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217790    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.217826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217820    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.218119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217884    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.218119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.217925    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.218119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.218043    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.218119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.218083    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.219835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.219817    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.220327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.220311    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.229545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.229481    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.229817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.229794    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhntf\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-kube-api-access-dhntf\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.300775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.300736    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:18.320394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320357    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2kr\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-kube-api-access-dt2kr\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.320515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320417    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c630ce-a087-471a-8916-1d97307638fd-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.320515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320476    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.320598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320524    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c630ce-a087-471a-8916-1d97307638fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.320651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320618    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.320783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320765    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.320886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320861    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.320969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320917    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.321036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.320992    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.321368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.321176    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.321479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.321398    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.321479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.321361    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.321594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.321534    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.321807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.321787    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c630ce-a087-471a-8916-1d97307638fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.323827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.323799    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c630ce-a087-471a-8916-1d97307638fd-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.323925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.323910    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.331944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.331920    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.332120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.332097    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2kr\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-kube-api-access-dt2kr\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.438321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.438284    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:18.649659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.649629    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"]
Oct 30 21:28:18.650092 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:28:18.650067    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c66e81d_a2b5_4c15_9bfc_9e65d3b0a887.slice/crio-665d6aac6a84f16a84f7b99d7db35f98a361f4843a92cad43b3c6b6d2db88d84 WatchSource:0}: Error finding container 665d6aac6a84f16a84f7b99d7db35f98a361f4843a92cad43b3c6b6d2db88d84: Status 404 returned error can't find the container with id 665d6aac6a84f16a84f7b99d7db35f98a361f4843a92cad43b3c6b6d2db88d84
Oct 30 21:28:18.653007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.652976    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:18.653098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.653044    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:18.653098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.653076    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:18.767308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.767280    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"]
Oct 30 21:28:18.768768 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:28:18.768740    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c630ce_a087_471a_8916_1d97307638fd.slice/crio-379ab3c89508313c6c48d15f196d80af6a756dc9428694611d010fa950f4143e WatchSource:0}: Error finding container 379ab3c89508313c6c48d15f196d80af6a756dc9428694611d010fa950f4143e: Status 404 returned error can't find the container with id 379ab3c89508313c6c48d15f196d80af6a756dc9428694611d010fa950f4143e
Oct 30 21:28:18.772455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.772420    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:18.772564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.772504    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:18.772564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:18.772552    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:19.508116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:19.508079    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" event={"ID":"b7c630ce-a087-471a-8916-1d97307638fd","Type":"ContainerStarted","Data":"3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631"}
Oct 30 21:28:19.508582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:19.508142    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" event={"ID":"b7c630ce-a087-471a-8916-1d97307638fd","Type":"ContainerStarted","Data":"379ab3c89508313c6c48d15f196d80af6a756dc9428694611d010fa950f4143e"}
Oct 30 21:28:19.510081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:19.510052    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" event={"ID":"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887","Type":"ContainerStarted","Data":"6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f"}
Oct 30 21:28:19.510211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:19.510105    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" event={"ID":"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887","Type":"ContainerStarted","Data":"665d6aac6a84f16a84f7b99d7db35f98a361f4843a92cad43b3c6b6d2db88d84"}
Oct 30 21:28:19.572658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:19.572563    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" podStartSLOduration=1.572544143 podStartE2EDuration="1.572544143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:19.568064066 +0000 UTC m=+1607.917791462" watchObservedRunningTime="2026-10-30 21:28:19.572544143 +0000 UTC m=+1607.922271541"
Oct 30 21:28:19.608904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:19.606085    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" podStartSLOduration=2.6060663699999997 podStartE2EDuration="2.60606637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:19.601934994 +0000 UTC m=+1607.951662388" watchObservedRunningTime="2026-10-30 21:28:19.60606637 +0000 UTC m=+1607.955793767"
Oct 30 21:28:20.313696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:20.304478    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:20.321694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:20.317844    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:28:20.321694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:20.317912    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 21:28:20.469739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:20.464733    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:20.469739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:20.467954    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:28:20.469739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:20.468033    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 21:28:21.301365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:21.301333    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:28:21.301856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:21.301829    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 21:28:21.441695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:21.441468    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:28:21.441695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:21.441549    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 21:28:22.170818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:22.170775    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:22.184275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:22.184242    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:22.302058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:22.302028    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:28:22.302423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:22.302080    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 21:28:22.455774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:22.455061    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:28:22.455774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:22.455124    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 21:28:22.781438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:22.781351    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:28:23.301584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:23.301546    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:28:23.301830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:23.301604    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 21:28:23.450697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:23.448206    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:28:23.450697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:23.448328    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 21:28:24.301758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:24.301724    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:28:24.301943 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:24.301776    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 21:28:24.448696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:24.448479    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 21:28:24.448696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:24.448537    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 21:28:25.070775 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:28:25.070505    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:28:25.310747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:25.310711    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:28:25.311156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:25.311120    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 30 21:28:25.493663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:25.493626    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:25.494465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:25.494382    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:25.495850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:25.495554    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:28:26.308128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:26.308097    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:26.308569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:26.308548    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:26.311003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:26.310977    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:28:28.995356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:28.995321    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"]
Oct 30 21:28:29.001811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.001788    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.013359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.013329    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-904993\"/\"kube-root-ca.crt\""
Oct 30 21:28:29.019033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.019011    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"]
Oct 30 21:28:29.021126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.021105    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-904993\"/\"default-dockercfg-qpg8c\""
Oct 30 21:28:29.021233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.021186    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-904993\"/\"auth-test-model-904993-kserve-self-signed-certs\""
Oct 30 21:28:29.025387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.025352    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-dshm\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.025504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.025405    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-model-cache\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.025575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.025547    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tmp-dir\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.025645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.025587    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-home\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.025645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.025618    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxq8\" (UniqueName: \"kubernetes.io/projected/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-kube-api-access-zxxq8\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.025645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.025638    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tls-certs\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.115622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.115578    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-904993\"/\"openshift-service-ca.crt\""
Oct 30 21:28:29.126974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.126943    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-dshm\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.127141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.127011    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-model-cache\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.127197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.127173    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tmp-dir\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.127239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.127227    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-home\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.127294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.127279    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxq8\" (UniqueName: \"kubernetes.io/projected/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-kube-api-access-zxxq8\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.127356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.127312    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tls-certs\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.127459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.127441    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-model-cache\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.127554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.127531    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tmp-dir\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.127631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.127580    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-home\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.129473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.129449    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-dshm\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.129858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.129841    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tls-certs\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.142581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.142554    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxq8\" (UniqueName: \"kubernetes.io/projected/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-kube-api-access-zxxq8\") pod \"auth-test-model-904993-kserve-54f98948c7-hwhxp\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") " pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.319602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.318962    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:29.789202 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:28:29.789167    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfc9d4ba_db00_477a_92a3_f0ed1a5c2e5b.slice/crio-e526b18c0f3ed894ae7909a249399f0d81cb509a839b5b8a154879ceaa3b6579 WatchSource:0}: Error finding container e526b18c0f3ed894ae7909a249399f0d81cb509a839b5b8a154879ceaa3b6579: Status 404 returned error can't find the container with id e526b18c0f3ed894ae7909a249399f0d81cb509a839b5b8a154879ceaa3b6579
Oct 30 21:28:29.794147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:29.794124    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"]
Oct 30 21:28:30.736481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:30.736445    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp" event={"ID":"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b","Type":"ContainerStarted","Data":"4cf4f9bc163b25fa8b8282016a91eeca56f2b1e632bc95556f70e717d7b07c51"}
Oct 30 21:28:30.736481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:30.736486    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp" event={"ID":"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b","Type":"ContainerStarted","Data":"e526b18c0f3ed894ae7909a249399f0d81cb509a839b5b8a154879ceaa3b6579"}
Oct 30 21:28:30.764678 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:30.764622    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp" podStartSLOduration=2.7646082610000002 podStartE2EDuration="2.764608261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:30.762442142 +0000 UTC m=+1619.112169539" watchObservedRunningTime="2026-10-30 21:28:30.764608261 +0000 UTC m=+1619.114335657"
Oct 30 21:28:31.320502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:31.320462    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:39.320552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:39.320520    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:39.334247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:39.334219    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:39.953889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:39.953861    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:28:48.061428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.061390    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"]
Oct 30 21:28:48.067828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.067794    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.089103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.089072    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-726a18-openshift-default-dockercfg-b45mt\""
Oct 30 21:28:48.095134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.095093    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"]
Oct 30 21:28:48.135415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.135374    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.135620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.135508    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-token\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.135713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.135649    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-podinfo\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.135811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.135790    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-certs\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.135885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.135863    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-credential-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.135942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.135900    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbf88\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-kube-api-access-sbf88\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.136002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.135986    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-envoy\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.136057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.136039    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-data\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.136111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.136065    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istiod-ca-cert\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.237382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237344    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-token\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.237564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237447    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-podinfo\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.237611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237567    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-certs\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.237654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237609    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-credential-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.237654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237632    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbf88\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-kube-api-access-sbf88\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.237806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237761    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-envoy\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.237920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237815    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-data\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.237920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237841    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istiod-ca-cert\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.238031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.237978    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.238090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.238045    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-certs\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.238150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.238123    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-credential-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.238343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.238319    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-data\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.238416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.238323    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.238608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.238543    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istiod-ca-cert\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.240193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.240165    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-envoy\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.240461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.240441    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-podinfo\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.254377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.254351    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbf88\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-kube-api-access-sbf88\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.260124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.260099    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-token\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.385050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.384971    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:48.720133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.717265    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:48.720133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.717367    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:48.720133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.717412    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:48.721695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:48.721616    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"]
Oct 30 21:28:49.055849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:49.055756    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" event={"ID":"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a","Type":"ContainerStarted","Data":"866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8"}
Oct 30 21:28:49.055849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:49.055814    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" event={"ID":"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a","Type":"ContainerStarted","Data":"0c4c2b8ab0c4993763c375442fc53493bffb11551a9df8fe4b49e37996d5edcf"}
Oct 30 21:28:49.104640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:49.104584    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podStartSLOduration=2.104569321 podStartE2EDuration="2.104569321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:49.09684941 +0000 UTC m=+1637.446576828" watchObservedRunningTime="2026-10-30 21:28:49.104569321 +0000 UTC m=+1637.454296799"
Oct 30 21:28:49.385278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:49.385192    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:49.387313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:49.387285    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:28:49.387443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:49.387339    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 21:28:50.386277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:50.386242    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:28:50.386708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:50.386301    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 21:28:51.386178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:51.386143    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:28:51.386366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:51.386208    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 21:28:51.839139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:51.839096    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:28:52.391695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:52.386579    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:28:52.391695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:52.386657    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 21:28:53.406046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:53.405923    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:28:53.406046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:53.406011    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 21:28:54.386051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:54.386013    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:28:54.386263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:54.386071    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 21:28:55.045931 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:28:55.045893    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:28:55.386092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:55.386007    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:28:55.386344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:55.386313    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 21:28:56.575623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:56.575585    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:56.576173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:56.576144    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:56.577142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:56.577111    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:28:58.817579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.817547    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"]
Oct 30 21:28:58.822514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.822494    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.842141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.842115    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6ed261-openshift-default-dockercfg-6stmn\""
Oct 30 21:28:58.849879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.849849    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"]
Oct 30 21:28:58.957792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.957751    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-credential-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.977222 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.977185    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae502d4-f440-470b-a8f9-957c672d6745-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.977631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.977603    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.977991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.977967    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-data\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.978136 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.978119    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-certs\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.978336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.978318    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae502d4-f440-470b-a8f9-957c672d6745-istio-podinfo\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.978478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.978460    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f4fh\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-kube-api-access-2f4fh\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.978660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.978642    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-istio-token\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:58.978873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:58.978855    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-envoy\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081055    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-credential-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081141    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae502d4-f440-470b-a8f9-957c672d6745-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081243    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081404    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-data\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081444    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-certs\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081477    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-credential-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081510    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae502d4-f440-470b-a8f9-957c672d6745-istio-podinfo\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081558    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f4fh\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-kube-api-access-2f4fh\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081585    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081620    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-istio-token\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081705    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-envoy\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.081753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081746    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-data\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.082013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.081992    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae502d4-f440-470b-a8f9-957c672d6745-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.082152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.082124    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-certs\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.084262 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.084240    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-envoy\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.084438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.084416    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae502d4-f440-470b-a8f9-957c672d6745-istio-podinfo\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.092090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.092061    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-istio-token\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.092230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.092150    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f4fh\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-kube-api-access-2f4fh\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.139468 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.139439    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:28:59.473803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.473775    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"]
Oct 30 21:28:59.474905 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:28:59.474883    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae502d4_f440_470b_a8f9_957c672d6745.slice/crio-f65f1005b0dc0c13ab6d547e22d2a1408731d5d81888b8233054f2a687074fa2 WatchSource:0}: Error finding container f65f1005b0dc0c13ab6d547e22d2a1408731d5d81888b8233054f2a687074fa2: Status 404 returned error can't find the container with id f65f1005b0dc0c13ab6d547e22d2a1408731d5d81888b8233054f2a687074fa2
Oct 30 21:28:59.477951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.477917    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:59.478045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.477991    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:28:59.478045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:28:59.478021    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:29:00.242951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:00.242911    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" event={"ID":"2ae502d4-f440-470b-a8f9-957c672d6745","Type":"ContainerStarted","Data":"1fdc524fc15d2c5f27cb864fe49f773bee01b24fb57b6ebd9f552af03bd88c5f"}
Oct 30 21:29:00.243439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:00.242968    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" event={"ID":"2ae502d4-f440-470b-a8f9-957c672d6745","Type":"ContainerStarted","Data":"f65f1005b0dc0c13ab6d547e22d2a1408731d5d81888b8233054f2a687074fa2"}
Oct 30 21:29:00.276678 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:00.276624    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" podStartSLOduration=2.276609228 podStartE2EDuration="2.276609228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:00.274467391 +0000 UTC m=+1648.624194787" watchObservedRunningTime="2026-10-30 21:29:00.276609228 +0000 UTC m=+1648.626336623"
Oct 30 21:29:01.140353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:01.140313    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:29:01.142110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:01.142083    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:29:01.142256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:01.142142    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 21:29:02.140947 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:02.140910    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:29:02.141458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:02.140977    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 21:29:02.295168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:02.295124    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"]
Oct 30 21:29:03.140187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:03.140154    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:29:03.140394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:03.140212    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 21:29:04.139897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:04.139863    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:29:04.140382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:04.139924    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 21:29:05.140307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:05.140267    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:29:05.140864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:05.140351    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 21:29:06.140604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:06.140568    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 30 21:29:06.141269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:06.140627    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 30 21:29:07.146027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:07.145997    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:29:07.146515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:07.146470    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:29:07.147232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:07.147212    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:29:17.348192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.348099    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck" podUID="3004e018-fc29-48a1-b7a8-a71fb646cf63" containerName="main" containerID="cri-o://3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037" gracePeriod=45
Oct 30 21:29:17.384225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.384191    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck" podUID="3004e018-fc29-48a1-b7a8-a71fb646cf63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.192:8000/ready\": dial tcp 10.133.0.192:8000: connect: connection refused"
Oct 30 21:29:17.717490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.717157    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:29:17.805680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.805633    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3004e018-fc29-48a1-b7a8-a71fb646cf63-kube-api-access-5g92v\" (UniqueName: \"kubernetes.io/projected/3004e018-fc29-48a1-b7a8-a71fb646cf63-kube-api-access-5g92v\") pod \"3004e018-fc29-48a1-b7a8-a71fb646cf63\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") "
Oct 30 21:29:17.805870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.805749    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3004e018-fc29-48a1-b7a8-a71fb646cf63-tls-certs\" (UniqueName: \"kubernetes.io/secret/3004e018-fc29-48a1-b7a8-a71fb646cf63-tls-certs\") pod \"3004e018-fc29-48a1-b7a8-a71fb646cf63\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") "
Oct 30 21:29:17.805870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.805843    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-dshm\") pod \"3004e018-fc29-48a1-b7a8-a71fb646cf63\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") "
Oct 30 21:29:17.806065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.806048    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-model-cache\") pod \"3004e018-fc29-48a1-b7a8-a71fb646cf63\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") "
Oct 30 21:29:17.806164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.806095    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-home\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-home\") pod \"3004e018-fc29-48a1-b7a8-a71fb646cf63\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") "
Oct 30 21:29:17.806164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.806124    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-tmp-dir\") pod \"3004e018-fc29-48a1-b7a8-a71fb646cf63\" (UID: \"3004e018-fc29-48a1-b7a8-a71fb646cf63\") "
Oct 30 21:29:17.806494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.806411    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-model-cache" pod "3004e018-fc29-48a1-b7a8-a71fb646cf63" (UID: "3004e018-fc29-48a1-b7a8-a71fb646cf63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:17.806494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.806453    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-home" pod "3004e018-fc29-48a1-b7a8-a71fb646cf63" (UID: "3004e018-fc29-48a1-b7a8-a71fb646cf63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:17.806838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.806620    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-tmp-dir" pod "3004e018-fc29-48a1-b7a8-a71fb646cf63" (UID: "3004e018-fc29-48a1-b7a8-a71fb646cf63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:17.807094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.807069    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:17.807214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.807099    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:17.807214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.807143    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:17.809449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.809385    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3004e018-fc29-48a1-b7a8-a71fb646cf63-tls-certs" pod "3004e018-fc29-48a1-b7a8-a71fb646cf63" (UID: "3004e018-fc29-48a1-b7a8-a71fb646cf63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:29:17.809449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.809417    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3004e018-fc29-48a1-b7a8-a71fb646cf63-kube-api-access-5g92v" pod "3004e018-fc29-48a1-b7a8-a71fb646cf63" (UID: "3004e018-fc29-48a1-b7a8-a71fb646cf63"). InnerVolumeSpecName "kube-api-access-5g92v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:17.810877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.810850    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-dshm" pod "3004e018-fc29-48a1-b7a8-a71fb646cf63" (UID: "3004e018-fc29-48a1-b7a8-a71fb646cf63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:17.908972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.908936    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g92v\" (UniqueName: \"kubernetes.io/projected/3004e018-fc29-48a1-b7a8-a71fb646cf63-kube-api-access-5g92v\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:17.908972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.908972    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3004e018-fc29-48a1-b7a8-a71fb646cf63-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:17.909174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:17.908989    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3004e018-fc29-48a1-b7a8-a71fb646cf63-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:18.423744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.423561    4633 generic.go:356] "Generic (PLEG): container finished" podID="3004e018-fc29-48a1-b7a8-a71fb646cf63" containerID="3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037" exitCode=0
Oct 30 21:29:18.423744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.423611    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"
Oct 30 21:29:18.423744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.423642    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck" event={"ID":"3004e018-fc29-48a1-b7a8-a71fb646cf63","Type":"ContainerDied","Data":"3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037"}
Oct 30 21:29:18.423744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.423715    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck" event={"ID":"3004e018-fc29-48a1-b7a8-a71fb646cf63","Type":"ContainerDied","Data":"a51c6c76ed791f8ab893fe05f15f958573527ebdb445d9e28d086cc6c45235be"}
Oct 30 21:29:18.423744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.423739    4633 scope.go:122] "RemoveContainer" containerID="3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037"
Oct 30 21:29:18.441871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.441848    4633 scope.go:122] "RemoveContainer" containerID="3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037"
Oct 30 21:29:18.442217 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:29:18.442199    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037\": container with ID starting with 3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037 not found: ID does not exist" containerID="3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037"
Oct 30 21:29:18.442270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.442230    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037"} err="failed to get container status \"3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037\": rpc error: code = NotFound desc = could not find container \"3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037\": container with ID starting with 3a303fff7c291064af82022095def26b5060f35158db453361341f15207dd037 not found: ID does not exist"
Oct 30 21:29:18.479998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.479966    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"]
Oct 30 21:29:18.494073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:18.494037    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15e79c/auth-test-model-15e79c-kserve-598d599d77-bqfck"]
Oct 30 21:29:19.793407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:19.793373    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3004e018-fc29-48a1-b7a8-a71fb646cf63" path="/var/lib/kubelet/pods/3004e018-fc29-48a1-b7a8-a71fb646cf63/volumes"
Oct 30 21:29:25.236697 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:29:25.234107    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:29:36.707761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:36.707725    4633 generic.go:356] "Generic (PLEG): container finished" podID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerID="ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901" exitCode=0
Oct 30 21:29:36.708229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:36.707800    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" event={"ID":"3ff05b49-76c1-45dc-9f4d-3d07699c12d7","Type":"ContainerDied","Data":"ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901"}
Oct 30 21:29:36.708305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:36.708287    4633 scope.go:122] "RemoveContainer" containerID="ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901"
Oct 30 21:29:36.710395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:36.710366    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:29:36.710495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:36.710436    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:29:36.710495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:36.710465    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:29:37.743482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:37.743446    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" event={"ID":"3ff05b49-76c1-45dc-9f4d-3d07699c12d7","Type":"ContainerStarted","Data":"db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2"}
Oct 30 21:29:38.204802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:38.204769    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:29:38.212876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:38.212836    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:29:38.213192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:38.213160    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 21:29:38.688255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:38.688220    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:29:39.205697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:39.203719    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:29:39.205697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:39.203805    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 21:29:40.202979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:40.202947    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:29:40.203145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:40.202995    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 21:29:41.203779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:41.203732    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:29:41.204223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:41.203802    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 21:29:42.204258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:42.204217    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:29:42.205067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:42.205005    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 21:29:43.203463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:43.203430    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 30 21:29:43.203655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:43.203483    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 30 21:29:44.207574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:44.207546    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:29:44.208873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:44.208852    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:29:55.016898 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:29:55.016861    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:29:59.024459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:29:59.024424    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:30:00.408696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.403791    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"]
Oct 30 21:30:00.408696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.405047    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3004e018-fc29-48a1-b7a8-a71fb646cf63" containerName="main"
Oct 30 21:30:00.408696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.405081    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3004e018-fc29-48a1-b7a8-a71fb646cf63" containerName="main"
Oct 30 21:30:00.408696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.405415    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3004e018-fc29-48a1-b7a8-a71fb646cf63" containerName="main"
Oct 30 21:30:00.657158 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.629984    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:00.657158 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.647078    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:30:00.874027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.873994    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"]
Oct 30 21:30:00.882855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.882827    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:00.890227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.890202    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"]
Oct 30 21:30:00.904489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.904457    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cdf722-e445-45ec-869c-b4bc81f1f369-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg\" (UID: \"35cdf722-e445-45ec-869c-b4bc81f1f369\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:00.904715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.904565    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b\" (UID: \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:00.904715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.904607    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km644\" (UniqueName: \"kubernetes.io/projected/35cdf722-e445-45ec-869c-b4bc81f1f369-kube-api-access-km644\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg\" (UID: \"35cdf722-e445-45ec-869c-b4bc81f1f369\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:00.904873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.904852    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qh4\" (UniqueName: \"kubernetes.io/projected/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-kube-api-access-b5qh4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b\" (UID: \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:00.921438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.916462    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"]
Oct 30 21:30:00.924795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.924770    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"]
Oct 30 21:30:00.924997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.924982    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"]
Oct 30 21:30:00.925102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:00.924894    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:01.006456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.006418    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqql\" (UniqueName: \"kubernetes.io/projected/dd40b0c6-af91-45a9-a836-22f961cff95e-kube-api-access-hmqql\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt\" (UID: \"dd40b0c6-af91-45a9-a836-22f961cff95e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:01.006646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.006571    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qh4\" (UniqueName: \"kubernetes.io/projected/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-kube-api-access-b5qh4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b\" (UID: \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:01.006736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.006645    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40b0c6-af91-45a9-a836-22f961cff95e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt\" (UID: \"dd40b0c6-af91-45a9-a836-22f961cff95e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:01.006850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.006833    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cdf722-e445-45ec-869c-b4bc81f1f369-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg\" (UID: \"35cdf722-e445-45ec-869c-b4bc81f1f369\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:01.006916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.006897    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b\" (UID: \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:01.006978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.006929    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km644\" (UniqueName: \"kubernetes.io/projected/35cdf722-e445-45ec-869c-b4bc81f1f369-kube-api-access-km644\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg\" (UID: \"35cdf722-e445-45ec-869c-b4bc81f1f369\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:01.007542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.007517    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cdf722-e445-45ec-869c-b4bc81f1f369-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg\" (UID: \"35cdf722-e445-45ec-869c-b4bc81f1f369\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:01.007641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.007518    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b\" (UID: \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:01.035748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.035656    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qh4\" (UniqueName: \"kubernetes.io/projected/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-kube-api-access-b5qh4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b\" (UID: \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:01.039578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.039541    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km644\" (UniqueName: \"kubernetes.io/projected/35cdf722-e445-45ec-869c-b4bc81f1f369-kube-api-access-km644\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg\" (UID: \"35cdf722-e445-45ec-869c-b4bc81f1f369\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:01.108587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.108496    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqql\" (UniqueName: \"kubernetes.io/projected/dd40b0c6-af91-45a9-a836-22f961cff95e-kube-api-access-hmqql\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt\" (UID: \"dd40b0c6-af91-45a9-a836-22f961cff95e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:01.108771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.108694    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40b0c6-af91-45a9-a836-22f961cff95e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt\" (UID: \"dd40b0c6-af91-45a9-a836-22f961cff95e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:01.109553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.109528    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40b0c6-af91-45a9-a836-22f961cff95e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt\" (UID: \"dd40b0c6-af91-45a9-a836-22f961cff95e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:01.140081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.140030    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqql\" (UniqueName: \"kubernetes.io/projected/dd40b0c6-af91-45a9-a836-22f961cff95e-kube-api-access-hmqql\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt\" (UID: \"dd40b0c6-af91-45a9-a836-22f961cff95e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:01.211270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.210885    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:01.241189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.241151    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:01.246994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.246964    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:01.996306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:01.996277    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"]
Oct 30 21:30:02.212904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:02.212872    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"]
Oct 30 21:30:02.257594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:02.257542    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" event={"ID":"dd40b0c6-af91-45a9-a836-22f961cff95e","Type":"ContainerStarted","Data":"ffdb1ca6b0c6c13c37d1f67a1dce4e0e3f3a50cf35bc3a2b71369ce7887fc69c"}
Oct 30 21:30:02.261086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:02.261051    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" event={"ID":"35cdf722-e445-45ec-869c-b4bc81f1f369","Type":"ContainerStarted","Data":"a1282c987951b059ab3ab5e13aca25c8041273311c597ad0fcf4b2c658c5500c"}
Oct 30 21:30:02.315411 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:30:02.315373    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf37dc64_6cf5_4c57_9f2f_6327aa101c74.slice/crio-e3a4f4d654e342866488e3988a7dd5ccb188e1fc73316b2e1600e8ed40f39b8e WatchSource:0}: Error finding container e3a4f4d654e342866488e3988a7dd5ccb188e1fc73316b2e1600e8ed40f39b8e: Status 404 returned error can't find the container with id e3a4f4d654e342866488e3988a7dd5ccb188e1fc73316b2e1600e8ed40f39b8e
Oct 30 21:30:02.316087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:02.316054    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"]
Oct 30 21:30:03.270658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:03.270623    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" event={"ID":"cf37dc64-6cf5-4c57-9f2f-6327aa101c74","Type":"ContainerStarted","Data":"8126f5c26b7f76daa34bca01874254e2d145fbb1315c154991afd59a56e13586"}
Oct 30 21:30:03.271129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:03.270729    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" event={"ID":"cf37dc64-6cf5-4c57-9f2f-6327aa101c74","Type":"ContainerStarted","Data":"e3a4f4d654e342866488e3988a7dd5ccb188e1fc73316b2e1600e8ed40f39b8e"}
Oct 30 21:30:03.272570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:03.272542    4633 generic.go:356] "Generic (PLEG): container finished" podID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerID="0dcf4d5ed0d1d9a1a267e697993c6988291b6d9282d1a17b4271feff13fcbc02" exitCode=60
Oct 30 21:30:03.272726 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:03.272693    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" event={"ID":"35cdf722-e445-45ec-869c-b4bc81f1f369","Type":"ContainerDied","Data":"0dcf4d5ed0d1d9a1a267e697993c6988291b6d9282d1a17b4271feff13fcbc02"}
Oct 30 21:30:03.272993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:03.272978    4633 scope.go:122] "RemoveContainer" containerID="0dcf4d5ed0d1d9a1a267e697993c6988291b6d9282d1a17b4271feff13fcbc02"
Oct 30 21:30:03.274923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:03.274901    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" event={"ID":"dd40b0c6-af91-45a9-a836-22f961cff95e","Type":"ContainerStarted","Data":"70e8be2f2d31eee2475d90f96fa7ee42721398f2a09a8a5f7f29d96cadbbd0d9"}
Oct 30 21:30:03.316630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:03.316581    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" podStartSLOduration=3.316564916 podStartE2EDuration="3.316564916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.312495912 +0000 UTC m=+1711.662223309" watchObservedRunningTime="2026-10-30 21:30:03.316564916 +0000 UTC m=+1711.666292311"
Oct 30 21:30:03.467319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:03.463134    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" podStartSLOduration=3.463109407 podStartE2EDuration="3.463109407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.448364997 +0000 UTC m=+1711.798092395" watchObservedRunningTime="2026-10-30 21:30:03.463109407 +0000 UTC m=+1711.812836803"
Oct 30 21:30:04.284975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:04.284943    4633 generic.go:356] "Generic (PLEG): container finished" podID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerID="70e8be2f2d31eee2475d90f96fa7ee42721398f2a09a8a5f7f29d96cadbbd0d9" exitCode=60
Oct 30 21:30:04.285456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:04.285030    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" event={"ID":"dd40b0c6-af91-45a9-a836-22f961cff95e","Type":"ContainerDied","Data":"70e8be2f2d31eee2475d90f96fa7ee42721398f2a09a8a5f7f29d96cadbbd0d9"}
Oct 30 21:30:04.285456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:04.285307    4633 scope.go:122] "RemoveContainer" containerID="70e8be2f2d31eee2475d90f96fa7ee42721398f2a09a8a5f7f29d96cadbbd0d9"
Oct 30 21:30:04.287081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:04.287045    4633 generic.go:356] "Generic (PLEG): container finished" podID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerID="8126f5c26b7f76daa34bca01874254e2d145fbb1315c154991afd59a56e13586" exitCode=60
Oct 30 21:30:04.287216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:04.287096    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" event={"ID":"cf37dc64-6cf5-4c57-9f2f-6327aa101c74","Type":"ContainerDied","Data":"8126f5c26b7f76daa34bca01874254e2d145fbb1315c154991afd59a56e13586"}
Oct 30 21:30:04.287387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:04.287367    4633 scope.go:122] "RemoveContainer" containerID="8126f5c26b7f76daa34bca01874254e2d145fbb1315c154991afd59a56e13586"
Oct 30 21:30:04.290470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:04.290446    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" event={"ID":"35cdf722-e445-45ec-869c-b4bc81f1f369","Type":"ContainerStarted","Data":"0192b41d59832f57c19cb44cee963a1128b662b11dfb60c4877e24ae960fecbc"}
Oct 30 21:30:04.365139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:04.365078    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" podStartSLOduration=4.365058646 podStartE2EDuration="4.365058646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.360644676 +0000 UTC m=+1712.710372072" watchObservedRunningTime="2026-10-30 21:30:04.365058646 +0000 UTC m=+1712.714786023"
Oct 30 21:30:05.366812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.366779    4633 generic.go:356] "Generic (PLEG): container finished" podID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerID="0192b41d59832f57c19cb44cee963a1128b662b11dfb60c4877e24ae960fecbc" exitCode=60
Oct 30 21:30:05.367323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.367018    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" event={"ID":"35cdf722-e445-45ec-869c-b4bc81f1f369","Type":"ContainerDied","Data":"0192b41d59832f57c19cb44cee963a1128b662b11dfb60c4877e24ae960fecbc"}
Oct 30 21:30:05.367323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.367087    4633 scope.go:122] "RemoveContainer" containerID="0dcf4d5ed0d1d9a1a267e697993c6988291b6d9282d1a17b4271feff13fcbc02"
Oct 30 21:30:05.367800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.367780    4633 scope.go:122] "RemoveContainer" containerID="0192b41d59832f57c19cb44cee963a1128b662b11dfb60c4877e24ae960fecbc"
Oct 30 21:30:05.368340 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:05.368314    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg_odh-ai-gateway-infra(35cdf722-e445-45ec-869c-b4bc81f1f369)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369"
Oct 30 21:30:05.385575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.385538    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" event={"ID":"dd40b0c6-af91-45a9-a836-22f961cff95e","Type":"ContainerStarted","Data":"b72bdc22550fa4f167dc1713e25ee95c054ac2f3cb4f50a24ec77e7c075c61f4"}
Oct 30 21:30:05.388196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.388166    4633 generic.go:356] "Generic (PLEG): container finished" podID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerID="102e85aeb60c2e9f14671893fca4c0c303814d6f862b537f88e16e5231ee3f3c" exitCode=60
Oct 30 21:30:05.388326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.388253    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" event={"ID":"cf37dc64-6cf5-4c57-9f2f-6327aa101c74","Type":"ContainerDied","Data":"102e85aeb60c2e9f14671893fca4c0c303814d6f862b537f88e16e5231ee3f3c"}
Oct 30 21:30:05.388546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.388529    4633 scope.go:122] "RemoveContainer" containerID="102e85aeb60c2e9f14671893fca4c0c303814d6f862b537f88e16e5231ee3f3c"
Oct 30 21:30:05.388924 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:05.388902    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b_odh-ai-gateway-infra(cf37dc64-6cf5-4c57-9f2f-6327aa101c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74"
Oct 30 21:30:05.395030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:05.395000    4633 scope.go:122] "RemoveContainer" containerID="8126f5c26b7f76daa34bca01874254e2d145fbb1315c154991afd59a56e13586"
Oct 30 21:30:06.398629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:06.398600    4633 scope.go:122] "RemoveContainer" containerID="0192b41d59832f57c19cb44cee963a1128b662b11dfb60c4877e24ae960fecbc"
Oct 30 21:30:06.399271 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:06.399064    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg_odh-ai-gateway-infra(35cdf722-e445-45ec-869c-b4bc81f1f369)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369"
Oct 30 21:30:06.400451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:06.400429    4633 generic.go:356] "Generic (PLEG): container finished" podID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerID="b72bdc22550fa4f167dc1713e25ee95c054ac2f3cb4f50a24ec77e7c075c61f4" exitCode=60
Oct 30 21:30:06.400612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:06.400519    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" event={"ID":"dd40b0c6-af91-45a9-a836-22f961cff95e","Type":"ContainerDied","Data":"b72bdc22550fa4f167dc1713e25ee95c054ac2f3cb4f50a24ec77e7c075c61f4"}
Oct 30 21:30:06.400612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:06.400579    4633 scope.go:122] "RemoveContainer" containerID="70e8be2f2d31eee2475d90f96fa7ee42721398f2a09a8a5f7f29d96cadbbd0d9"
Oct 30 21:30:06.400859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:06.400843    4633 scope.go:122] "RemoveContainer" containerID="b72bdc22550fa4f167dc1713e25ee95c054ac2f3cb4f50a24ec77e7c075c61f4"
Oct 30 21:30:06.401242 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:06.401216    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt_odh-ai-gateway-infra(dd40b0c6-af91-45a9-a836-22f961cff95e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e"
Oct 30 21:30:06.402596 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:06.402579    4633 scope.go:122] "RemoveContainer" containerID="102e85aeb60c2e9f14671893fca4c0c303814d6f862b537f88e16e5231ee3f3c"
Oct 30 21:30:06.403018 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:06.402992    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b_odh-ai-gateway-infra(cf37dc64-6cf5-4c57-9f2f-6327aa101c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74"
Oct 30 21:30:07.413767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:07.413734    4633 scope.go:122] "RemoveContainer" containerID="b72bdc22550fa4f167dc1713e25ee95c054ac2f3cb4f50a24ec77e7c075c61f4"
Oct 30 21:30:07.414282 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:07.414107    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt_odh-ai-gateway-infra(dd40b0c6-af91-45a9-a836-22f961cff95e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e"
Oct 30 21:30:17.784245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:17.784211    4633 scope.go:122] "RemoveContainer" containerID="0192b41d59832f57c19cb44cee963a1128b662b11dfb60c4877e24ae960fecbc"
Oct 30 21:30:18.640967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:18.640931    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" event={"ID":"35cdf722-e445-45ec-869c-b4bc81f1f369","Type":"ContainerStarted","Data":"540a9539e4748cf66b30d81f48b4d2b7670cd2265bca630f0b927a5606f827a2"}
Oct 30 21:30:19.653116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:19.653078    4633 generic.go:356] "Generic (PLEG): container finished" podID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerID="540a9539e4748cf66b30d81f48b4d2b7670cd2265bca630f0b927a5606f827a2" exitCode=60
Oct 30 21:30:19.653545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:19.653151    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" event={"ID":"35cdf722-e445-45ec-869c-b4bc81f1f369","Type":"ContainerDied","Data":"540a9539e4748cf66b30d81f48b4d2b7670cd2265bca630f0b927a5606f827a2"}
Oct 30 21:30:19.653545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:19.653225    4633 scope.go:122] "RemoveContainer" containerID="0192b41d59832f57c19cb44cee963a1128b662b11dfb60c4877e24ae960fecbc"
Oct 30 21:30:19.653741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:19.653722    4633 scope.go:122] "RemoveContainer" containerID="540a9539e4748cf66b30d81f48b4d2b7670cd2265bca630f0b927a5606f827a2"
Oct 30 21:30:19.654102 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:19.654080    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg_odh-ai-gateway-infra(35cdf722-e445-45ec-869c-b4bc81f1f369)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369"
Oct 30 21:30:19.755720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:19.751723    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"]
Oct 30 21:30:19.784376 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:19.784344    4633 scope.go:122] "RemoveContainer" containerID="b72bdc22550fa4f167dc1713e25ee95c054ac2f3cb4f50a24ec77e7c075c61f4"
Oct 30 21:30:19.784558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:19.784406    4633 scope.go:122] "RemoveContainer" containerID="102e85aeb60c2e9f14671893fca4c0c303814d6f862b537f88e16e5231ee3f3c"
Oct 30 21:30:20.665049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:20.665005    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" event={"ID":"cf37dc64-6cf5-4c57-9f2f-6327aa101c74","Type":"ContainerStarted","Data":"27214c0df445523697b57822b094303112d6a5756d850a88f683158704f255fa"}
Oct 30 21:30:20.669754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:20.669726    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" event={"ID":"dd40b0c6-af91-45a9-a836-22f961cff95e","Type":"ContainerStarted","Data":"c9f9edcc8a02d44bc8103b4c22a8fd99ead69689f83303bf7c51731e2cf266a8"}
Oct 30 21:30:20.878659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:20.878627    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:20.994117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:20.994079    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cdf722-e445-45ec-869c-b4bc81f1f369-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cdf722-e445-45ec-869c-b4bc81f1f369-maas-api-service-ca\") pod \"35cdf722-e445-45ec-869c-b4bc81f1f369\" (UID: \"35cdf722-e445-45ec-869c-b4bc81f1f369\") "
Oct 30 21:30:20.994349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:20.994290    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cdf722-e445-45ec-869c-b4bc81f1f369-kube-api-access-km644\" (UniqueName: \"kubernetes.io/projected/35cdf722-e445-45ec-869c-b4bc81f1f369-kube-api-access-km644\") pod \"35cdf722-e445-45ec-869c-b4bc81f1f369\" (UID: \"35cdf722-e445-45ec-869c-b4bc81f1f369\") "
Oct 30 21:30:20.994461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:20.994438    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cdf722-e445-45ec-869c-b4bc81f1f369-maas-api-service-ca" pod "35cdf722-e445-45ec-869c-b4bc81f1f369" (UID: "35cdf722-e445-45ec-869c-b4bc81f1f369"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:20.995100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:20.995076    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35cdf722-e445-45ec-869c-b4bc81f1f369-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:20.996914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:20.996890    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cdf722-e445-45ec-869c-b4bc81f1f369-kube-api-access-km644" pod "35cdf722-e445-45ec-869c-b4bc81f1f369" (UID: "35cdf722-e445-45ec-869c-b4bc81f1f369"). InnerVolumeSpecName "kube-api-access-km644". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:21.096525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.096489    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km644\" (UniqueName: \"kubernetes.io/projected/35cdf722-e445-45ec-869c-b4bc81f1f369-kube-api-access-km644\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:21.680391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.680358    4633 generic.go:356] "Generic (PLEG): container finished" podID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerID="27214c0df445523697b57822b094303112d6a5756d850a88f683158704f255fa" exitCode=60
Oct 30 21:30:21.680867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.680431    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" event={"ID":"cf37dc64-6cf5-4c57-9f2f-6327aa101c74","Type":"ContainerDied","Data":"27214c0df445523697b57822b094303112d6a5756d850a88f683158704f255fa"}
Oct 30 21:30:21.680867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.680493    4633 scope.go:122] "RemoveContainer" containerID="102e85aeb60c2e9f14671893fca4c0c303814d6f862b537f88e16e5231ee3f3c"
Oct 30 21:30:21.681020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.680988    4633 scope.go:122] "RemoveContainer" containerID="27214c0df445523697b57822b094303112d6a5756d850a88f683158704f255fa"
Oct 30 21:30:21.681442 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:21.681419    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b_odh-ai-gateway-infra(cf37dc64-6cf5-4c57-9f2f-6327aa101c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74"
Oct 30 21:30:21.682286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.682270    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"
Oct 30 21:30:21.682353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.682301    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg" event={"ID":"35cdf722-e445-45ec-869c-b4bc81f1f369","Type":"ContainerDied","Data":"a1282c987951b059ab3ab5e13aca25c8041273311c597ad0fcf4b2c658c5500c"}
Oct 30 21:30:21.684686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.684651    4633 generic.go:356] "Generic (PLEG): container finished" podID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerID="c9f9edcc8a02d44bc8103b4c22a8fd99ead69689f83303bf7c51731e2cf266a8" exitCode=60
Oct 30 21:30:21.684804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.684741    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" event={"ID":"dd40b0c6-af91-45a9-a836-22f961cff95e","Type":"ContainerDied","Data":"c9f9edcc8a02d44bc8103b4c22a8fd99ead69689f83303bf7c51731e2cf266a8"}
Oct 30 21:30:21.685227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.685208    4633 scope.go:122] "RemoveContainer" containerID="c9f9edcc8a02d44bc8103b4c22a8fd99ead69689f83303bf7c51731e2cf266a8"
Oct 30 21:30:21.685574 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:21.685551    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt_odh-ai-gateway-infra(dd40b0c6-af91-45a9-a836-22f961cff95e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e"
Oct 30 21:30:21.699843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.699812    4633 scope.go:122] "RemoveContainer" containerID="540a9539e4748cf66b30d81f48b4d2b7670cd2265bca630f0b927a5606f827a2"
Oct 30 21:30:21.707708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.707637    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"]
Oct 30 21:30:21.711334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.711316    4633 scope.go:122] "RemoveContainer" containerID="b72bdc22550fa4f167dc1713e25ee95c054ac2f3cb4f50a24ec77e7c075c61f4"
Oct 30 21:30:21.740550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.740513    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"]
Oct 30 21:30:21.746115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.746087    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"]
Oct 30 21:30:21.758298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.758266    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-27804d-29889930-59xbg"]
Oct 30 21:30:21.794328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:21.794299    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" path="/var/lib/kubelet/pods/35cdf722-e445-45ec-869c-b4bc81f1f369/volumes"
Oct 30 21:30:22.910245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:22.910216    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:22.971511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:22.971432    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:23.029999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.029953    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-kube-api-access-b5qh4\" (UniqueName: \"kubernetes.io/projected/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-kube-api-access-b5qh4\") pod \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\" (UID: \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\") "
Oct 30 21:30:23.030193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.030134    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-maas-api-service-ca\") pod \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\" (UID: \"cf37dc64-6cf5-4c57-9f2f-6327aa101c74\") "
Oct 30 21:30:23.030647 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.030603    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-maas-api-service-ca" pod "cf37dc64-6cf5-4c57-9f2f-6327aa101c74" (UID: "cf37dc64-6cf5-4c57-9f2f-6327aa101c74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:23.031405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.031372    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:23.033123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.033095    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-kube-api-access-b5qh4" pod "cf37dc64-6cf5-4c57-9f2f-6327aa101c74" (UID: "cf37dc64-6cf5-4c57-9f2f-6327aa101c74"). InnerVolumeSpecName "kube-api-access-b5qh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:23.132906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.132862    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd40b0c6-af91-45a9-a836-22f961cff95e-kube-api-access-hmqql\" (UniqueName: \"kubernetes.io/projected/dd40b0c6-af91-45a9-a836-22f961cff95e-kube-api-access-hmqql\") pod \"dd40b0c6-af91-45a9-a836-22f961cff95e\" (UID: \"dd40b0c6-af91-45a9-a836-22f961cff95e\") "
Oct 30 21:30:23.133083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.133048    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd40b0c6-af91-45a9-a836-22f961cff95e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40b0c6-af91-45a9-a836-22f961cff95e-maas-api-service-ca\") pod \"dd40b0c6-af91-45a9-a836-22f961cff95e\" (UID: \"dd40b0c6-af91-45a9-a836-22f961cff95e\") "
Oct 30 21:30:23.133860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.133821    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd40b0c6-af91-45a9-a836-22f961cff95e-maas-api-service-ca" pod "dd40b0c6-af91-45a9-a836-22f961cff95e" (UID: "dd40b0c6-af91-45a9-a836-22f961cff95e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:23.139590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.135167    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd40b0c6-af91-45a9-a836-22f961cff95e-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:23.139590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.135205    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5qh4\" (UniqueName: \"kubernetes.io/projected/cf37dc64-6cf5-4c57-9f2f-6327aa101c74-kube-api-access-b5qh4\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:23.139590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.136577    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd40b0c6-af91-45a9-a836-22f961cff95e-kube-api-access-hmqql" pod "dd40b0c6-af91-45a9-a836-22f961cff95e" (UID: "dd40b0c6-af91-45a9-a836-22f961cff95e"). InnerVolumeSpecName "kube-api-access-hmqql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:23.271753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.267588    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmqql\" (UniqueName: \"kubernetes.io/projected/dd40b0c6-af91-45a9-a836-22f961cff95e-kube-api-access-hmqql\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:23.710770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.710739    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"
Oct 30 21:30:23.715728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.711318    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt" event={"ID":"dd40b0c6-af91-45a9-a836-22f961cff95e","Type":"ContainerDied","Data":"ffdb1ca6b0c6c13c37d1f67a1dce4e0e3f3a50cf35bc3a2b71369ce7887fc69c"}
Oct 30 21:30:23.715728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.711436    4633 scope.go:122] "RemoveContainer" containerID="c9f9edcc8a02d44bc8103b4c22a8fd99ead69689f83303bf7c51731e2cf266a8"
Oct 30 21:30:23.715728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.714008    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b" event={"ID":"cf37dc64-6cf5-4c57-9f2f-6327aa101c74","Type":"ContainerDied","Data":"e3a4f4d654e342866488e3988a7dd5ccb188e1fc73316b2e1600e8ed40f39b8e"}
Oct 30 21:30:23.715728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.714143    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"
Oct 30 21:30:23.731108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.731081    4633 scope.go:122] "RemoveContainer" containerID="27214c0df445523697b57822b094303112d6a5756d850a88f683158704f255fa"
Oct 30 21:30:23.756761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.756729    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"]
Oct 30 21:30:23.768321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.768292    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889930-vwfwt"]
Oct 30 21:30:23.795741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.795702    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" path="/var/lib/kubelet/pods/dd40b0c6-af91-45a9-a836-22f961cff95e/volumes"
Oct 30 21:30:23.796249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.796227    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"]
Oct 30 21:30:23.800373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:23.800346    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889930-bh57b"]
Oct 30 21:30:25.018851 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:25.018818    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:30:25.795206 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:25.795174    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" path="/var/lib/kubelet/pods/cf37dc64-6cf5-4c57-9f2f-6327aa101c74/volumes"
Oct 30 21:30:48.851331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:30:48.851240    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:30:55.018447 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:30:55.018402    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:31:13.030938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:13.030898    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"]
Oct 30 21:31:13.035145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:13.035118    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4a41fe-fm854"]
Oct 30 21:31:13.794237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:13.794203    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42235461-f97b-49c4-b280-15ab4ce97199" path="/var/lib/kubelet/pods/42235461-f97b-49c4-b280-15ab4ce97199/volumes"
Oct 30 21:31:25.213775 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:31:25.193291    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:31:32.446402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:32.446365    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:31:32.451049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:32.451019    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:31:32.470390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:32.470354    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:31:32.484347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:32.484311    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:31:34.782862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:34.782825    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"]
Oct 30 21:31:41.682709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:41.682639    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:31:42.130788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:42.130640    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"]
Oct 30 21:31:47.476826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:47.476791    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"]
Oct 30 21:31:49.812124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:49.812080    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf" podUID="1271668c-67d5-410e-ae88-b44237083f30" containerName="main" containerID="cri-o://91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c" gracePeriod=45
Oct 30 21:31:50.489163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.489134    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:31:50.590450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.590413    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1271668c-67d5-410e-ae88-b44237083f30-kube-api-access-vf7xh\" (UniqueName: \"kubernetes.io/projected/1271668c-67d5-410e-ae88-b44237083f30-kube-api-access-vf7xh\") pod \"1271668c-67d5-410e-ae88-b44237083f30\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") "
Oct 30 21:31:50.590634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.590616    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-dshm\") pod \"1271668c-67d5-410e-ae88-b44237083f30\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") "
Oct 30 21:31:50.590741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.590714    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-home\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-home\") pod \"1271668c-67d5-410e-ae88-b44237083f30\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") "
Oct 30 21:31:50.590811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.590757    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1271668c-67d5-410e-ae88-b44237083f30-tls-certs\" (UniqueName: \"kubernetes.io/secret/1271668c-67d5-410e-ae88-b44237083f30-tls-certs\") pod \"1271668c-67d5-410e-ae88-b44237083f30\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") "
Oct 30 21:31:50.590872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.590850    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-model-cache\") pod \"1271668c-67d5-410e-ae88-b44237083f30\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") "
Oct 30 21:31:50.590932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.590918    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-tmp-dir\") pod \"1271668c-67d5-410e-ae88-b44237083f30\" (UID: \"1271668c-67d5-410e-ae88-b44237083f30\") "
Oct 30 21:31:50.591034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.590997    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-home" pod "1271668c-67d5-410e-ae88-b44237083f30" (UID: "1271668c-67d5-410e-ae88-b44237083f30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:50.591234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.591210    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-model-cache" pod "1271668c-67d5-410e-ae88-b44237083f30" (UID: "1271668c-67d5-410e-ae88-b44237083f30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:50.591234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.591222    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-tmp-dir" pod "1271668c-67d5-410e-ae88-b44237083f30" (UID: "1271668c-67d5-410e-ae88-b44237083f30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:50.591917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.591894    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:50.591917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.591915    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:50.592107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.591924    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:50.593152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.593125    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1271668c-67d5-410e-ae88-b44237083f30-kube-api-access-vf7xh" pod "1271668c-67d5-410e-ae88-b44237083f30" (UID: "1271668c-67d5-410e-ae88-b44237083f30"). InnerVolumeSpecName "kube-api-access-vf7xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:50.593250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.593171    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-dshm" pod "1271668c-67d5-410e-ae88-b44237083f30" (UID: "1271668c-67d5-410e-ae88-b44237083f30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:50.593250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.593184    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1271668c-67d5-410e-ae88-b44237083f30-tls-certs" pod "1271668c-67d5-410e-ae88-b44237083f30" (UID: "1271668c-67d5-410e-ae88-b44237083f30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:50.693441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.693407    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1271668c-67d5-410e-ae88-b44237083f30-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:50.693441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.693437    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1271668c-67d5-410e-ae88-b44237083f30-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:50.693441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.693446    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf7xh\" (UniqueName: \"kubernetes.io/projected/1271668c-67d5-410e-ae88-b44237083f30-kube-api-access-vf7xh\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:50.763516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.763425    4633 generic.go:356] "Generic (PLEG): container finished" podID="1271668c-67d5-410e-ae88-b44237083f30" containerID="91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c" exitCode=0
Oct 30 21:31:50.763516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.763477    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"
Oct 30 21:31:50.763764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.763512    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf" event={"ID":"1271668c-67d5-410e-ae88-b44237083f30","Type":"ContainerDied","Data":"91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c"}
Oct 30 21:31:50.763764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.763561    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf" event={"ID":"1271668c-67d5-410e-ae88-b44237083f30","Type":"ContainerDied","Data":"f0a680fc180c1e311722bc620a4403fd3f474a2955ff6d72a81f8dc1e9daf7b9"}
Oct 30 21:31:50.763764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.763579    4633 scope.go:122] "RemoveContainer" containerID="91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c"
Oct 30 21:31:50.775949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.775931    4633 scope.go:122] "RemoveContainer" containerID="91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c"
Oct 30 21:31:50.776212 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:31:50.776187    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c\": container with ID starting with 91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c not found: ID does not exist" containerID="91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c"
Oct 30 21:31:50.776259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.776224    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c"} err="failed to get container status \"91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c\": rpc error: code = NotFound desc = could not find container \"91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c\": container with ID starting with 91e060e9632c07479c36c51714fdc1ec78c1b2c41f67fe34119bdfedf481e73c not found: ID does not exist"
Oct 30 21:31:50.790923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.790891    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"]
Oct 30 21:31:50.796344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:50.796320    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf"]
Oct 30 21:31:51.411781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:51.411743    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-b30a42/auth-test-model-b30a42-kserve-7ccd8cbf97-hbszf" podUID="1271668c-67d5-410e-ae88-b44237083f30" containerName="main" probeResult="failure" output="Get \"https://10.133.0.187:8000/ready\": context deadline exceeded"
Oct 30 21:31:51.868315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:51.868229    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1271668c-67d5-410e-ae88-b44237083f30" path="/var/lib/kubelet/pods/1271668c-67d5-410e-ae88-b44237083f30/volumes"
Oct 30 21:31:55.076038 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:31:55.075982    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:31:57.179103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.179058    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t" podUID="d648c9dd-5abf-460a-9d76-af50203d5df3" containerName="main" containerID="cri-o://a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247" gracePeriod=45
Oct 30 21:31:57.489316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.489290    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:31:57.543732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.543664    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-home\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-home\") pod \"d648c9dd-5abf-460a-9d76-af50203d5df3\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") "
Oct 30 21:31:57.543732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.543735    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d648c9dd-5abf-460a-9d76-af50203d5df3-kube-api-access-psrrz\" (UniqueName: \"kubernetes.io/projected/d648c9dd-5abf-460a-9d76-af50203d5df3-kube-api-access-psrrz\") pod \"d648c9dd-5abf-460a-9d76-af50203d5df3\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") "
Oct 30 21:31:57.544014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.543758    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-model-cache\") pod \"d648c9dd-5abf-460a-9d76-af50203d5df3\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") "
Oct 30 21:31:57.544084 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.544009    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-tmp-dir\") pod \"d648c9dd-5abf-460a-9d76-af50203d5df3\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") "
Oct 30 21:31:57.544084 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.544041    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-dshm\") pod \"d648c9dd-5abf-460a-9d76-af50203d5df3\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") "
Oct 30 21:31:57.544302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.544225    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d648c9dd-5abf-460a-9d76-af50203d5df3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d648c9dd-5abf-460a-9d76-af50203d5df3-tls-certs\") pod \"d648c9dd-5abf-460a-9d76-af50203d5df3\" (UID: \"d648c9dd-5abf-460a-9d76-af50203d5df3\") "
Oct 30 21:31:57.544542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.544518    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-model-cache" pod "d648c9dd-5abf-460a-9d76-af50203d5df3" (UID: "d648c9dd-5abf-460a-9d76-af50203d5df3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:57.545330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.545111    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:57.545453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.545187    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-home" pod "d648c9dd-5abf-460a-9d76-af50203d5df3" (UID: "d648c9dd-5abf-460a-9d76-af50203d5df3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:57.545781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.545757    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-tmp-dir" pod "d648c9dd-5abf-460a-9d76-af50203d5df3" (UID: "d648c9dd-5abf-460a-9d76-af50203d5df3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:57.547282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.547257    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d648c9dd-5abf-460a-9d76-af50203d5df3-tls-certs" pod "d648c9dd-5abf-460a-9d76-af50203d5df3" (UID: "d648c9dd-5abf-460a-9d76-af50203d5df3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:57.547433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.547414    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d648c9dd-5abf-460a-9d76-af50203d5df3-kube-api-access-psrrz" pod "d648c9dd-5abf-460a-9d76-af50203d5df3" (UID: "d648c9dd-5abf-460a-9d76-af50203d5df3"). InnerVolumeSpecName "kube-api-access-psrrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:57.547501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.547478    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-dshm" pod "d648c9dd-5abf-460a-9d76-af50203d5df3" (UID: "d648c9dd-5abf-460a-9d76-af50203d5df3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:31:57.648171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.648056    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:57.648171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.648093    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:57.648171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.648108    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d648c9dd-5abf-460a-9d76-af50203d5df3-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:57.648171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.648120    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d648c9dd-5abf-460a-9d76-af50203d5df3-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:57.648171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.648137    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psrrz\" (UniqueName: \"kubernetes.io/projected/d648c9dd-5abf-460a-9d76-af50203d5df3-kube-api-access-psrrz\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:57.959442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.959409    4633 generic.go:356] "Generic (PLEG): container finished" podID="d648c9dd-5abf-460a-9d76-af50203d5df3" containerID="a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247" exitCode=0
Oct 30 21:31:57.959608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.959461    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"
Oct 30 21:31:57.959608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.959488    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t" event={"ID":"d648c9dd-5abf-460a-9d76-af50203d5df3","Type":"ContainerDied","Data":"a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247"}
Oct 30 21:31:57.959608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.959520    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t" event={"ID":"d648c9dd-5abf-460a-9d76-af50203d5df3","Type":"ContainerDied","Data":"0967acb857a12a91cf1dff5fa92a5646c6e74b44e459c0fd62b51075a0ec5151"}
Oct 30 21:31:57.959608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.959536    4633 scope.go:122] "RemoveContainer" containerID="a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247"
Oct 30 21:31:57.970996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.970975    4633 scope.go:122] "RemoveContainer" containerID="a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247"
Oct 30 21:31:57.971278 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:31:57.971254    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247\": container with ID starting with a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247 not found: ID does not exist" containerID="a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247"
Oct 30 21:31:57.971380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.971284    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247"} err="failed to get container status \"a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247\": rpc error: code = NotFound desc = could not find container \"a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247\": container with ID starting with a440f9e055a4168a8ddbc1d4f7c8f25236eb18c6376dd765f1e620552e5dc247 not found: ID does not exist"
Oct 30 21:31:57.979991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.979956    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"]
Oct 30 21:31:57.985155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:57.985135    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-128d2f/auth-test-model-128d2f-kserve-868cc88fc7-qkw6t"]
Oct 30 21:31:59.793214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:31:59.793180    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d648c9dd-5abf-460a-9d76-af50203d5df3" path="/var/lib/kubelet/pods/d648c9dd-5abf-460a-9d76-af50203d5df3/volumes"
Oct 30 21:32:02.507351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:02.507304    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d" podUID="a41b8717-7029-4354-ae73-a6df0b7d6b7b" containerName="main" containerID="cri-o://e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314" gracePeriod=45
Oct 30 21:32:02.558462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:02.558407    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"]
Oct 30 21:32:02.558722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:02.558694    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy" containerID="cri-o://d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f" gracePeriod=30
Oct 30 21:32:02.565863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:02.565827    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"]
Oct 30 21:32:02.566090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:02.566065    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy" containerID="cri-o://d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e" gracePeriod=30
Oct 30 21:32:02.655742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:02.655663    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d" podUID="a41b8717-7029-4354-ae73-a6df0b7d6b7b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": dial tcp 10.133.0.193:8000: connect: connection refused"
Oct 30 21:32:02.950111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:02.950076    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.016947    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-home\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-home\") pod \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") "
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.016988    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-dshm\") pod \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") "
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.017159    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tmp-dir\") pod \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") "
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.017234    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tls-certs\") pod \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") "
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.017295    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-model-cache\") pod \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") "
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.017427    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tmp-dir" pod "a41b8717-7029-4354-ae73-a6df0b7d6b7b" (UID: "a41b8717-7029-4354-ae73-a6df0b7d6b7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.017448    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41b8717-7029-4354-ae73-a6df0b7d6b7b-kube-api-access-6b44g\" (UniqueName: \"kubernetes.io/projected/a41b8717-7029-4354-ae73-a6df0b7d6b7b-kube-api-access-6b44g\") pod \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\" (UID: \"a41b8717-7029-4354-ae73-a6df0b7d6b7b\") "
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.017612    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-model-cache" pod "a41b8717-7029-4354-ae73-a6df0b7d6b7b" (UID: "a41b8717-7029-4354-ae73-a6df0b7d6b7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.018049    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-home" pod "a41b8717-7029-4354-ae73-a6df0b7d6b7b" (UID: "a41b8717-7029-4354-ae73-a6df0b7d6b7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.018254    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.018271    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.021722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.018279    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.024044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.022814    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-dshm" pod "a41b8717-7029-4354-ae73-a6df0b7d6b7b" (UID: "a41b8717-7029-4354-ae73-a6df0b7d6b7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:03.024044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.022858    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41b8717-7029-4354-ae73-a6df0b7d6b7b-kube-api-access-6b44g" pod "a41b8717-7029-4354-ae73-a6df0b7d6b7b" (UID: "a41b8717-7029-4354-ae73-a6df0b7d6b7b"). InnerVolumeSpecName "kube-api-access-6b44g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:03.024044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.023059    4633 generic.go:356] "Generic (PLEG): container finished" podID="a41b8717-7029-4354-ae73-a6df0b7d6b7b" containerID="e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314" exitCode=0
Oct 30 21:32:03.024044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.023209    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d" event={"ID":"a41b8717-7029-4354-ae73-a6df0b7d6b7b","Type":"ContainerDied","Data":"e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314"}
Oct 30 21:32:03.024044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.023251    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d" event={"ID":"a41b8717-7029-4354-ae73-a6df0b7d6b7b","Type":"ContainerDied","Data":"224fa435abdf22551935d69914c93e65887959ce65977d1701bc61d54703064e"}
Oct 30 21:32:03.024044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.023276    4633 scope.go:122] "RemoveContainer" containerID="e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314"
Oct 30 21:32:03.024044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.023432    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"
Oct 30 21:32:03.024044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.023899    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tls-certs" pod "a41b8717-7029-4354-ae73-a6df0b7d6b7b" (UID: "a41b8717-7029-4354-ae73-a6df0b7d6b7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:03.061659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.061636    4633 scope.go:122] "RemoveContainer" containerID="e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314"
Oct 30 21:32:03.062254 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:03.062212    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314\": container with ID starting with e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314 not found: ID does not exist" containerID="e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314"
Oct 30 21:32:03.062379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.062252    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314"} err="failed to get container status \"e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314\": rpc error: code = NotFound desc = could not find container \"e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314\": container with ID starting with e8d2c087fc633aef5023a8a40478e3e8a76ee20545fe2dd421a59616bd124314 not found: ID does not exist"
Oct 30 21:32:03.120447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.120410    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b44g\" (UniqueName: \"kubernetes.io/projected/a41b8717-7029-4354-ae73-a6df0b7d6b7b-kube-api-access-6b44g\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.120447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.120440    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41b8717-7029-4354-ae73-a6df0b7d6b7b-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.120447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.120449    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41b8717-7029-4354-ae73-a6df0b7d6b7b-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.354957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.354922    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"]
Oct 30 21:32:03.360833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.360798    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6ed261/auth-test-model-6ed261-kserve-58dd98d645-wwd2d"]
Oct 30 21:32:03.801133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:03.801095    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41b8717-7029-4354-ae73-a6df0b7d6b7b" path="/var/lib/kubelet/pods/a41b8717-7029-4354-ae73-a6df0b7d6b7b/volumes"
Oct 30 21:32:05.988988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:05.988922    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp" podUID="cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.196:8000/ready\": context deadline exceeded"
Oct 30 21:32:06.091754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:06.071260    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:07.933968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:07.932050    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:32:07.972817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:07.972788    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.013572    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-kube-api-access-nnmx4\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-kube-api-access-nnmx4\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.013657    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-istio-token\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-istio-token\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.013722    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-certs\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.013782    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-kube-api-access-jzsrv\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-kube-api-access-jzsrv\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.013857    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-socket\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.013972    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-envoy\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014033    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-credential-socket\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014078    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-token\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-token\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014109    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f3a996-275d-4731-81b7-d8e8c240f429-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f3a996-275d-4731-81b7-d8e8c240f429-istiod-ca-cert\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014181    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5182b128-53a7-48f4-915e-f808a22318db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5182b128-53a7-48f4-915e-f808a22318db-istio-podinfo\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014363    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-socket\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014413    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-certs\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014538    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5182b128-53a7-48f4-915e-f808a22318db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5182b128-53a7-48f4-915e-f808a22318db-istiod-ca-cert\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014613    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-credential-socket\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014725    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-data\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014777    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-envoy\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014822    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-data\") pod \"5182b128-53a7-48f4-915e-f808a22318db\" (UID: \"5182b128-53a7-48f4-915e-f808a22318db\") "
Oct 30 21:32:08.016282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.014882    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-podinfo\") pod \"a6f3a996-275d-4731-81b7-d8e8c240f429\" (UID: \"a6f3a996-275d-4731-81b7-d8e8c240f429\") "
Oct 30 21:32:08.018559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.018529    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f3a996-275d-4731-81b7-d8e8c240f429-istiod-ca-cert" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:08.019580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.019538    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-certs" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.020216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.020190    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-credential-socket" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.020422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.020401    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-socket" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.020537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.020387    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-socket" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.020861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.020841    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-data" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.021003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.020870    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5182b128-53a7-48f4-915e-f808a22318db-istiod-ca-cert" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:08.021095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.021052    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-certs" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.021373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.021352    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-credential-socket" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.023237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.022954    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5182b128-53a7-48f4-915e-f808a22318db-istio-podinfo" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:08.023237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.023192    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-data" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.023521 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.023487    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-envoy" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.024565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.024542    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-token" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:08.030241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.030191    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-kube-api-access-nnmx4" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "kube-api-access-nnmx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:08.030241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.030266    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-podinfo" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:08.031420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.031394    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-kube-api-access-jzsrv" pod "a6f3a996-275d-4731-81b7-d8e8c240f429" (UID: "a6f3a996-275d-4731-81b7-d8e8c240f429"). InnerVolumeSpecName "kube-api-access-jzsrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:08.032156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.032134    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-istio-token" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:08.032244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.032212    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-envoy" pod "5182b128-53a7-48f4-915e-f808a22318db" (UID: "5182b128-53a7-48f4-915e-f808a22318db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:08.120065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.119986    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120015    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120025    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120034    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120044    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnmx4\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-kube-api-access-nnmx4\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120053    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5182b128-53a7-48f4-915e-f808a22318db-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120062    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120071    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzsrv\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-kube-api-access-jzsrv\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120082    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120090    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120098    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120107    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f3a996-275d-4731-81b7-d8e8c240f429-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120115    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f3a996-275d-4731-81b7-d8e8c240f429-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120123    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5182b128-53a7-48f4-915e-f808a22318db-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120131    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5182b128-53a7-48f4-915e-f808a22318db-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120141    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120148    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5182b128-53a7-48f4-915e-f808a22318db-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.120457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.120156    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f3a996-275d-4731-81b7-d8e8c240f429-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:08.496918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.496880    4633 generic.go:356] "Generic (PLEG): container finished" podID="5182b128-53a7-48f4-915e-f808a22318db" containerID="d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e" exitCode=0
Oct 30 21:32:08.497121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.496961    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"
Oct 30 21:32:08.497121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.496968    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" event={"ID":"5182b128-53a7-48f4-915e-f808a22318db","Type":"ContainerDied","Data":"d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e"}
Oct 30 21:32:08.497121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.497063    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw" event={"ID":"5182b128-53a7-48f4-915e-f808a22318db","Type":"ContainerDied","Data":"a893503fb4d9fee5ff64dd8e783a44be7c8378dd811ce47ed73336ae5ec81142"}
Oct 30 21:32:08.497121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.497095    4633 scope.go:122] "RemoveContainer" containerID="d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e"
Oct 30 21:32:08.498896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.498867    4633 generic.go:356] "Generic (PLEG): container finished" podID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerID="d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f" exitCode=0
Oct 30 21:32:08.499036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.498937    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"
Oct 30 21:32:08.499036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.498970    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" event={"ID":"a6f3a996-275d-4731-81b7-d8e8c240f429","Type":"ContainerDied","Data":"d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f"}
Oct 30 21:32:08.499036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.499011    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7" event={"ID":"a6f3a996-275d-4731-81b7-d8e8c240f429","Type":"ContainerDied","Data":"ff9c8c730f1a4c92196260e93ec0d5af943b90e342f86fcea0af1aafe97a468d"}
Oct 30 21:32:08.509568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.509549    4633 scope.go:122] "RemoveContainer" containerID="d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e"
Oct 30 21:32:08.509908 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:08.509890    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e\": container with ID starting with d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e not found: ID does not exist" containerID="d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e"
Oct 30 21:32:08.509971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.509916    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e"} err="failed to get container status \"d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e\": rpc error: code = NotFound desc = could not find container \"d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e\": container with ID starting with d0ff8b2be989d22fe763002d6af2431e399b2f2f939f9751d84e082ecb3b010e not found: ID does not exist"
Oct 30 21:32:08.509971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.509936    4633 scope.go:122] "RemoveContainer" containerID="d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f"
Oct 30 21:32:08.522801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.522782    4633 scope.go:122] "RemoveContainer" containerID="d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f"
Oct 30 21:32:08.523262 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:08.523242    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f\": container with ID starting with d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f not found: ID does not exist" containerID="d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f"
Oct 30 21:32:08.523344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.523271    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f"} err="failed to get container status \"d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f\": rpc error: code = NotFound desc = could not find container \"d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f\": container with ID starting with d23649a9abb9db5e8316f3f8566e474eda8a839e1bf5623c2b5a0af324db637f not found: ID does not exist"
Oct 30 21:32:08.526431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.526395    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"]
Oct 30 21:32:08.531213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.531193    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-8dhrw"]
Oct 30 21:32:08.541744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.541714    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"]
Oct 30 21:32:08.549507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:08.549480    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-pkfw7"]
Oct 30 21:32:09.539952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.539912    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"]
Oct 30 21:32:09.541719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.541693    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1271668c-67d5-410e-ae88-b44237083f30" containerName="main"
Oct 30 21:32:09.541935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.541920    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1271668c-67d5-410e-ae88-b44237083f30" containerName="main"
Oct 30 21:32:09.542046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.542036    4633 cpu_manager.go:414] "containerMap: removing container" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:09.542125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.542115    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:09.542210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.542201    4633 cpu_manager.go:414] "containerMap: removing container" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:09.542295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.542286    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:09.542374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.542365    4633 cpu_manager.go:414] "containerMap: removing container" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:09.543818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.543775    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:09.544013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.543998    4633 cpu_manager.go:414] "containerMap: removing container" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:09.544143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544130    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:09.544252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544241    4633 cpu_manager.go:414] "containerMap: removing container" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:09.544345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544333    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:09.544444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544433    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a41b8717-7029-4354-ae73-a6df0b7d6b7b" containerName="main"
Oct 30 21:32:09.544526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544515    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41b8717-7029-4354-ae73-a6df0b7d6b7b" containerName="main"
Oct 30 21:32:09.544622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544611    4633 cpu_manager.go:414] "containerMap: removing container" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:09.544774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544759    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:09.544878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544866    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d648c9dd-5abf-460a-9d76-af50203d5df3" containerName="main"
Oct 30 21:32:09.544983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.544970    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d648c9dd-5abf-460a-9d76-af50203d5df3" containerName="main"
Oct 30 21:32:09.545076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.545065    4633 cpu_manager.go:414] "containerMap: removing container" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy"
Oct 30 21:32:09.545163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.545152    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy"
Oct 30 21:32:09.545265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.545255    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy"
Oct 30 21:32:09.545348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.545338    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy"
Oct 30 21:32:09.545733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.545699    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1271668c-67d5-410e-ae88-b44237083f30" containerName="main"
Oct 30 21:32:09.545888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.545874    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:09.545979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.545969    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:09.546086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546075    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:09.546167 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546156    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:09.546239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546231    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:09.546312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546302    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41b8717-7029-4354-ae73-a6df0b7d6b7b" containerName="main"
Oct 30 21:32:09.546395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546387    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:09.546550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546537    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:09.546637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546627    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d648c9dd-5abf-460a-9d76-af50203d5df3" containerName="main"
Oct 30 21:32:09.546737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546727    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5182b128-53a7-48f4-915e-f808a22318db" containerName="istio-proxy"
Oct 30 21:32:09.546833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546824    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" containerName="istio-proxy"
Oct 30 21:32:09.546917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.546909    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:09.577979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.563916    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"]
Oct 30 21:32:09.577979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.564065    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.577979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.570287    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-fb2b58-serving-cert\""
Oct 30 21:32:09.705699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.700933    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkbs\" (UniqueName: \"kubernetes.io/projected/c8f26023-41db-4a13-b6be-912de216ccc5-kube-api-access-fpkbs\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.705699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.701559    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f26023-41db-4a13-b6be-912de216ccc5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.705699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.701743    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-metrics-certs\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.705699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.701985    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.794648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.794555    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5182b128-53a7-48f4-915e-f808a22318db" path="/var/lib/kubelet/pods/5182b128-53a7-48f4-915e-f808a22318db/volumes"
Oct 30 21:32:09.795326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.795297    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f3a996-275d-4731-81b7-d8e8c240f429" path="/var/lib/kubelet/pods/a6f3a996-275d-4731-81b7-d8e8c240f429/volumes"
Oct 30 21:32:09.803127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.803095    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkbs\" (UniqueName: \"kubernetes.io/projected/c8f26023-41db-4a13-b6be-912de216ccc5-kube-api-access-fpkbs\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.803286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.803161    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f26023-41db-4a13-b6be-912de216ccc5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.803286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.803193    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-metrics-certs\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.803411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.803338    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.803472 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:09.803451    4633 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-serving-cert: secret "maas-api-e2e-extmod-a-fb2b58-serving-cert" not found
Oct 30 21:32:09.803530 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:09.803518    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls podName:c8f26023-41db-4a13-b6be-912de216ccc5 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:10.303497418 +0000 UTC m=+1838.653224794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls") pod "maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z" (UID: "c8f26023-41db-4a13-b6be-912de216ccc5") : secret "maas-api-e2e-extmod-a-fb2b58-serving-cert" not found
Oct 30 21:32:09.804069 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.804047    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f26023-41db-4a13-b6be-912de216ccc5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.806561 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.806537    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-metrics-certs\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:09.811568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:09.811546    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkbs\" (UniqueName: \"kubernetes.io/projected/c8f26023-41db-4a13-b6be-912de216ccc5-kube-api-access-fpkbs\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:10.319447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:10.316778    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:10.352828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:10.348272    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:10.490911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:10.490871    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:10.782457 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:10.782411    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f26023_41db_4a13_b6be_912de216ccc5.slice/crio-e8b826255fbf288bbaa77524c66dbc4230ddd3ff6bc414f53a1793358d412d9b WatchSource:0}: Error finding container e8b826255fbf288bbaa77524c66dbc4230ddd3ff6bc414f53a1793358d412d9b: Status 404 returned error can't find the container with id e8b826255fbf288bbaa77524c66dbc4230ddd3ff6bc414f53a1793358d412d9b
Oct 30 21:32:10.782916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:10.782897    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"]
Oct 30 21:32:10.784564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:10.784539    4633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:32:11.628714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:11.628586    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z" event={"ID":"c8f26023-41db-4a13-b6be-912de216ccc5","Type":"ContainerStarted","Data":"382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65"}
Oct 30 21:32:11.629621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:11.629602    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:11.629844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:11.629824    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z" event={"ID":"c8f26023-41db-4a13-b6be-912de216ccc5","Type":"ContainerStarted","Data":"e8b826255fbf288bbaa77524c66dbc4230ddd3ff6bc414f53a1793358d412d9b"}
Oct 30 21:32:11.660342 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:11.660289    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z" podStartSLOduration=2.245496099 podStartE2EDuration="2.660274249s" totalImagesPullingTime="414.77815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:11.653729282 +0000 UTC m=+1840.003456683" watchObservedRunningTime="2026-10-30 21:32:11.660274249 +0000 UTC m=+1840.010001645"
Oct 30 21:32:14.708900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:14.708805    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"]
Oct 30 21:32:14.709379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:14.709050    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy" containerID="cri-o://d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78" gracePeriod=30
Oct 30 21:32:14.713650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:14.713608    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"]
Oct 30 21:32:14.714031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:14.714003    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy" containerID="cri-o://eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e" gracePeriod=30
Oct 30 21:32:14.719083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:14.719060    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"]
Oct 30 21:32:14.719288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:14.719265    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy" containerID="cri-o://cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633" gracePeriod=30
Oct 30 21:32:15.574380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.574341    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"]
Oct 30 21:32:15.575259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.575240    4633 cpu_manager.go:414] "containerMap: removing container" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:15.575324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.575261    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd40b0c6-af91-45a9-a836-22f961cff95e" containerName="cleanup"
Oct 30 21:32:15.575324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.575285    4633 cpu_manager.go:414] "containerMap: removing container" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:15.575324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.575290    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:15.575324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.575306    4633 cpu_manager.go:414] "containerMap: removing container" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:15.575324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.575311    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cdf722-e445-45ec-869c-b4bc81f1f369" containerName="cleanup"
Oct 30 21:32:15.575489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.575464    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf37dc64-6cf5-4c57-9f2f-6327aa101c74" containerName="cleanup"
Oct 30 21:32:15.579601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.579580    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.585046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.585021    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"kube-root-ca.crt\""
Oct 30 21:32:15.585167 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.585119    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-e2e--650b641a-dockercfg-s6bl5\""
Oct 30 21:32:15.585240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.585167    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"openshift-service-ca.crt\""
Oct 30 21:32:15.585740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.585026    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"routing-overlay\""
Oct 30 21:32:15.585856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.585787    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1ed97\""
Oct 30 21:32:15.585856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.585825    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-c07a671227da\""
Oct 30 21:32:15.591973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.591948    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"]
Oct 30 21:32:15.670023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.669991    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvnl\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-kube-api-access-mgvnl\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.670192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.670060    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.670192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.670147    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.670367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.670346    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.778698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.778615    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvnl\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-kube-api-access-mgvnl\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.779189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.778751    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.779189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.778805    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.779189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.779028    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.786708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.782867    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.786708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.783540    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.810699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.806522    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvnl\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-kube-api-access-mgvnl\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.820701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.812716    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:15.904706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:15.903976    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:16.331395 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:16.331362    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25c4119_4cef_4dc3_a72a_d0b38d7b38fa.slice/crio-168cf1521258e7ac4d50e6c3db32214e2d77903fa19d7196444436514465a1c0 WatchSource:0}: Error finding container 168cf1521258e7ac4d50e6c3db32214e2d77903fa19d7196444436514465a1c0: Status 404 returned error can't find the container with id 168cf1521258e7ac4d50e6c3db32214e2d77903fa19d7196444436514465a1c0
Oct 30 21:32:16.332430 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:16.332403    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"]
Oct 30 21:32:16.694142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:16.692605    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859" event={"ID":"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa","Type":"ContainerStarted","Data":"9ad1b1262bf20b9544d4208ba0f641328167c3e69c931cd3e348031a8a3abee1"}
Oct 30 21:32:16.694142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:16.692728    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859" event={"ID":"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa","Type":"ContainerStarted","Data":"168cf1521258e7ac4d50e6c3db32214e2d77903fa19d7196444436514465a1c0"}
Oct 30 21:32:16.694142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:16.693065    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:16.712094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:16.712029    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859" podStartSLOduration=1.712009151 podStartE2EDuration="1.712009151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:16.711429627 +0000 UTC m=+1845.061157036" watchObservedRunningTime="2026-10-30 21:32:16.712009151 +0000 UTC m=+1845.061736547"
Oct 30 21:32:18.650833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:18.650806    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:32:18.770995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:18.770955    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"]
Oct 30 21:32:19.866734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.856167    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"]
Oct 30 21:32:19.866734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.863711    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.879745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.879709    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"]
Oct 30 21:32:19.881252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.880415    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-02ecd7-openshift-default-dockercfg-drrk7\""
Oct 30 21:32:19.902358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.902315    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.902558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.902422    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmkxg\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-kube-api-access-xmkxg\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.902558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.902478    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.902749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.902727    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.902829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.902767    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.902907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.902843    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.902965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.902948    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.903029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.902979    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:19.906237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:19.904878    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b242af-fb91-4654-9b6a-975c80d2a871-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042017    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042099    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042166    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042324    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042374    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042469    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b242af-fb91-4654-9b6a-975c80d2a871-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042647    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042784    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkxg\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-kube-api-access-xmkxg\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.056795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.042839    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.069697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.065657    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b242af-fb91-4654-9b6a-975c80d2a871-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.082692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.071776    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.082692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.076841    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.082692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.077089    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.082692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.080874    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.301700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.179831    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.301700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.191274    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.301700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.235498    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.312363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.312325    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkxg\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-kube-api-access-xmkxg\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.340188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.340156    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:32:20.447092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.447064    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:32:20.450634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.450611    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476212    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-credential-socket\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476364    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-envoy\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476412    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-certs\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476468    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istiod-ca-cert\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476525    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-token\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-token\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476568    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-podinfo\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476596    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-kube-api-access-cvfmz\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-kube-api-access-cvfmz\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476651    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-socket\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.476764    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-data\") pod \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\" (UID: \"b0b1e6f9-5eae-4485-849b-2209b534f9eb\") "
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.477174    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-credential-socket" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.477432    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-certs" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.478227    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.478250    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.480625    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-envoy" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.480894    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-token" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.481067    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-socket" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.481288    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-data" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.481731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.481332    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-kube-api-access-cvfmz" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "kube-api-access-cvfmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:20.482889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.481799    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istiod-ca-cert" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:20.497026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.493795    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-podinfo" pod "b0b1e6f9-5eae-4485-849b-2209b534f9eb" (UID: "b0b1e6f9-5eae-4485-849b-2209b534f9eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:20.572783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.565107    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:20.579290 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579262    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-credential-socket\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.579385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579320    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47db1ff-15b0-4942-b38f-84e0c342659f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47db1ff-15b0-4942-b38f-84e0c342659f-istiod-ca-cert\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.579385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579360    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-certs\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.579504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579419    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-token\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-token\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.579504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579474    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-envoy\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.579610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579499    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-socket\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.579610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579546    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-socket\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.579610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579564    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-credential-socket" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.579610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579593    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-data\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.579858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579619    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-data\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.579858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579662    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-kube-api-access-l55qd\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-kube-api-access-l55qd\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.579858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579738    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-credential-socket\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.579858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579795    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-token\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-token\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.579858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579838    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-certs" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.580120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.579974    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-podinfo\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.580120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580028    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-kube-api-access-mp4nt\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-kube-api-access-mp4nt\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.580120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580100    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-socket" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.580292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580125    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istiod-ca-cert\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.580292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580173    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-podinfo\") pod \"f47db1ff-15b0-4942-b38f-84e0c342659f\" (UID: \"f47db1ff-15b0-4942-b38f-84e0c342659f\") "
Oct 30 21:32:20.580292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580238    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-data" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.580454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580286    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-envoy\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.580454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580366    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-certs\") pod \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\" (UID: \"108b52bf-e469-4ddf-80a7-5cace0eaf9fa\") "
Oct 30 21:32:20.580454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580444    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-socket" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.580920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.580897    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47db1ff-15b0-4942-b38f-84e0c342659f-istiod-ca-cert" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:20.581150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581129    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-data" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.581422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581399    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istiod-ca-cert" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:20.581585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581563    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581593    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581608    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581623    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581637    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581650    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvfmz\" (UniqueName: \"kubernetes.io/projected/b0b1e6f9-5eae-4485-849b-2209b534f9eb-kube-api-access-cvfmz\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581663    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581696    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581714    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f47db1ff-15b0-4942-b38f-84e0c342659f-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581744    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581757    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b1e6f9-5eae-4485-849b-2209b534f9eb-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581772    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581792    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581815    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.581987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.581837    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.585247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.585151    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-credential-socket" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.585247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.585155    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-certs" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.586132 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.586106    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-kube-api-access-l55qd" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "kube-api-access-l55qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:20.587496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.587455    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-token" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:20.590249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.590074    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-podinfo" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:20.590249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.590221    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-token" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:20.591296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.590654    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-envoy" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.591296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.590894    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-envoy" pod "108b52bf-e469-4ddf-80a7-5cace0eaf9fa" (UID: "108b52bf-e469-4ddf-80a7-5cace0eaf9fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:20.591573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.591542    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-podinfo" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:20.591814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.591777    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-kube-api-access-mp4nt" pod "f47db1ff-15b0-4942-b38f-84e0c342659f" (UID: "f47db1ff-15b0-4942-b38f-84e0c342659f"). InnerVolumeSpecName "kube-api-access-mp4nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:20.684090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684033    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4nt\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-kube-api-access-mp4nt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684095    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684111    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684128    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684144    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684158    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684173    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55qd\" (UniqueName: \"kubernetes.io/projected/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-kube-api-access-l55qd\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684188    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f47db1ff-15b0-4942-b38f-84e0c342659f-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684202    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f47db1ff-15b0-4942-b38f-84e0c342659f-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.684268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.684216    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108b52bf-e469-4ddf-80a7-5cace0eaf9fa-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:20.738414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.738378    4633 generic.go:356] "Generic (PLEG): container finished" podID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerID="d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78" exitCode=0
Oct 30 21:32:20.738606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.738432    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"
Oct 30 21:32:20.738606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.738460    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" event={"ID":"f47db1ff-15b0-4942-b38f-84e0c342659f","Type":"ContainerDied","Data":"d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78"}
Oct 30 21:32:20.738606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.738527    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c" event={"ID":"f47db1ff-15b0-4942-b38f-84e0c342659f","Type":"ContainerDied","Data":"1f242e554633f2d3227bdccbd1029438269d4e76574e3dae7614bbf16c4be724"}
Oct 30 21:32:20.738606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.738556    4633 scope.go:122] "RemoveContainer" containerID="d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78"
Oct 30 21:32:20.740612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.740584    4633 generic.go:356] "Generic (PLEG): container finished" podID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerID="cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633" exitCode=0
Oct 30 21:32:20.740788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.740621    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" event={"ID":"b0b1e6f9-5eae-4485-849b-2209b534f9eb","Type":"ContainerDied","Data":"cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633"}
Oct 30 21:32:20.740788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.740655    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"
Oct 30 21:32:20.740788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.740758    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc" event={"ID":"b0b1e6f9-5eae-4485-849b-2209b534f9eb","Type":"ContainerDied","Data":"533ed3f54f10ba865585a0ee9098d5fd75a0f81fe200f699b0101752b678b4da"}
Oct 30 21:32:20.742528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.742506    4633 generic.go:356] "Generic (PLEG): container finished" podID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerID="eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e" exitCode=0
Oct 30 21:32:20.742653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.742573    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" event={"ID":"108b52bf-e469-4ddf-80a7-5cace0eaf9fa","Type":"ContainerDied","Data":"eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e"}
Oct 30 21:32:20.742653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.742606    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7" event={"ID":"108b52bf-e469-4ddf-80a7-5cace0eaf9fa","Type":"ContainerDied","Data":"fd48f9b11eff74224516d2f01b6baca4e79dba4e52a9f41010d7807fd53ed851"}
Oct 30 21:32:20.742826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.742687    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"
Oct 30 21:32:20.785140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.785119    4633 scope.go:122] "RemoveContainer" containerID="d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78"
Oct 30 21:32:20.785463 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:20.785446    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78\": container with ID starting with d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78 not found: ID does not exist" containerID="d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78"
Oct 30 21:32:20.785557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.785477    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78"} err="failed to get container status \"d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78\": rpc error: code = NotFound desc = could not find container \"d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78\": container with ID starting with d864a43fb8b8824d64df19125a767d3d739b9628d65646b1a90f1a056acb3f78 not found: ID does not exist"
Oct 30 21:32:20.785557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.785502    4633 scope.go:122] "RemoveContainer" containerID="cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633"
Oct 30 21:32:20.796609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.796579    4633 scope.go:122] "RemoveContainer" containerID="cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633"
Oct 30 21:32:20.797000 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:20.796970    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633\": container with ID starting with cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633 not found: ID does not exist" containerID="cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633"
Oct 30 21:32:20.797101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.797002    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633"} err="failed to get container status \"cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633\": rpc error: code = NotFound desc = could not find container \"cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633\": container with ID starting with cd39b6c4277e72ad41c0250909e5747813f782bcbebff548723723da61697633 not found: ID does not exist"
Oct 30 21:32:20.797101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.797028    4633 scope.go:122] "RemoveContainer" containerID="eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e"
Oct 30 21:32:20.800926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.800903    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"]
Oct 30 21:32:20.806382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.806362    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-zrw8c"]
Oct 30 21:32:20.807930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.807916    4633 scope.go:122] "RemoveContainer" containerID="eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e"
Oct 30 21:32:20.808221 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:20.808204    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e\": container with ID starting with eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e not found: ID does not exist" containerID="eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e"
Oct 30 21:32:20.808280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.808228    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e"} err="failed to get container status \"eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e\": rpc error: code = NotFound desc = could not find container \"eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e\": container with ID starting with eabf632558b58f1c5fdd6d8f0c99014e0ac4582a64cf211e2b6b8fe05352974e not found: ID does not exist"
Oct 30 21:32:20.816523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.816499    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"]
Oct 30 21:32:20.821769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.821746    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-lwgf7"]
Oct 30 21:32:20.831816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.831744    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"]
Oct 30 21:32:20.834854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.834832    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-29rqc"]
Oct 30 21:32:20.882897 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:20.882865    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b242af_fb91_4654_9b6a_975c80d2a871.slice/crio-acd1a941a949593c56d441befa1a655b83ac5c67d0b04724fa44498123c064e1 WatchSource:0}: Error finding container acd1a941a949593c56d441befa1a655b83ac5c67d0b04724fa44498123c064e1: Status 404 returned error can't find the container with id acd1a941a949593c56d441befa1a655b83ac5c67d0b04724fa44498123c064e1
Oct 30 21:32:20.883314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.883055    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"]
Oct 30 21:32:20.886102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.886066    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:20.886182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.886134    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:20.886182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:20.886167    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:21.759096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:21.759063    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" event={"ID":"a9b242af-fb91-4654-9b6a-975c80d2a871","Type":"ContainerStarted","Data":"bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233"}
Oct 30 21:32:21.759308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:21.759112    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" event={"ID":"a9b242af-fb91-4654-9b6a-975c80d2a871","Type":"ContainerStarted","Data":"acd1a941a949593c56d441befa1a655b83ac5c67d0b04724fa44498123c064e1"}
Oct 30 21:32:21.805568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:21.805499    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podStartSLOduration=2.805478366 podStartE2EDuration="2.805478366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:21.785315263 +0000 UTC m=+1850.135042660" watchObservedRunningTime="2026-10-30 21:32:21.805478366 +0000 UTC m=+1850.155205763"
Oct 30 21:32:21.808250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:21.808223    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" path="/var/lib/kubelet/pods/108b52bf-e469-4ddf-80a7-5cace0eaf9fa/volumes"
Oct 30 21:32:21.808792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:21.808777    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" path="/var/lib/kubelet/pods/b0b1e6f9-5eae-4485-849b-2209b534f9eb/volumes"
Oct 30 21:32:21.809267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:21.809253    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" path="/var/lib/kubelet/pods/f47db1ff-15b0-4942-b38f-84e0c342659f/volumes"
Oct 30 21:32:22.568078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:22.568044    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:22.573650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:22.573622    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:22.767371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:22.767345    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:22.768278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:22.768259    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:32:25.016271 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:25.016227    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.933227    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"]
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934378    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy"
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934400    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy"
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934482    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy"
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934491    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy"
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934510    4633 cpu_manager.go:414] "containerMap: removing container" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy"
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934517    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy"
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934744    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47db1ff-15b0-4942-b38f-84e0c342659f" containerName="istio-proxy"
Oct 30 21:32:25.934760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934769    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b1e6f9-5eae-4485-849b-2209b534f9eb" containerName="istio-proxy"
Oct 30 21:32:25.935484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.934783    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="108b52bf-e469-4ddf-80a7-5cace0eaf9fa" containerName="istio-proxy"
Oct 30 21:32:25.942720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.940080    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"]
Oct 30 21:32:25.942720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.940209    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:25.984994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.983725    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-02ecd7\""
Oct 30 21:32:25.991779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:25.986921    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-02ecd7-dockercfg-r6fg6\""
Oct 30 21:32:26.117707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.117445    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsj7k\" (UniqueName: \"kubernetes.io/projected/2a701134-7e5f-4b2b-81f5-6f1d1925a863-kube-api-access-vsj7k\") pod \"payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g\" (UID: \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:26.173158 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.153588    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a701134-7e5f-4b2b-81f5-6f1d1925a863-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g\" (UID: \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:26.173343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.173231    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"]
Oct 30 21:32:26.201137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.201057    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:26.211441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.211322    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"]
Oct 30 21:32:26.263703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.260037    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a701134-7e5f-4b2b-81f5-6f1d1925a863-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g\" (UID: \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:26.263703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.260238    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsj7k\" (UniqueName: \"kubernetes.io/projected/2a701134-7e5f-4b2b-81f5-6f1d1925a863-kube-api-access-vsj7k\") pod \"payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g\" (UID: \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:26.263703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.261574    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a701134-7e5f-4b2b-81f5-6f1d1925a863-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g\" (UID: \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:26.272415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.272385    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsj7k\" (UniqueName: \"kubernetes.io/projected/2a701134-7e5f-4b2b-81f5-6f1d1925a863-kube-api-access-vsj7k\") pod \"payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g\" (UID: \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:26.305280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.305248    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:26.374695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.363935    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krx7f\" (UniqueName: \"kubernetes.io/projected/d73bf3bf-0b3d-417a-bf59-b86906557bad-kube-api-access-krx7f\") pod \"payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5\" (UID: \"d73bf3bf-0b3d-417a-bf59-b86906557bad\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:26.374695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.367120    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf3bf-0b3d-417a-bf59-b86906557bad-plugins-config-volume\") pod \"payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5\" (UID: \"d73bf3bf-0b3d-417a-bf59-b86906557bad\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:26.469157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.469117    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf3bf-0b3d-417a-bf59-b86906557bad-plugins-config-volume\") pod \"payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5\" (UID: \"d73bf3bf-0b3d-417a-bf59-b86906557bad\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:26.470333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.469424    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krx7f\" (UniqueName: \"kubernetes.io/projected/d73bf3bf-0b3d-417a-bf59-b86906557bad-kube-api-access-krx7f\") pod \"payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5\" (UID: \"d73bf3bf-0b3d-417a-bf59-b86906557bad\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:26.470333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.470285    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf3bf-0b3d-417a-bf59-b86906557bad-plugins-config-volume\") pod \"payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5\" (UID: \"d73bf3bf-0b3d-417a-bf59-b86906557bad\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:26.483365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.483341    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krx7f\" (UniqueName: \"kubernetes.io/projected/d73bf3bf-0b3d-417a-bf59-b86906557bad-kube-api-access-krx7f\") pod \"payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5\" (UID: \"d73bf3bf-0b3d-417a-bf59-b86906557bad\") " pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:26.532161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.532125    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:26.688076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.688045    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"]
Oct 30 21:32:26.702416 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:26.702353    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a701134_7e5f_4b2b_81f5_6f1d1925a863.slice/crio-689bdcb00ce0f18a27a719fefcb72cf447da6d0aa85e0faab9f22962f9fe0992 WatchSource:0}: Error finding container 689bdcb00ce0f18a27a719fefcb72cf447da6d0aa85e0faab9f22962f9fe0992: Status 404 returned error can't find the container with id 689bdcb00ce0f18a27a719fefcb72cf447da6d0aa85e0faab9f22962f9fe0992
Oct 30 21:32:26.830194 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:26.830167    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73bf3bf_0b3d_417a_bf59_b86906557bad.slice/crio-81be8b55f46d1c76e24e26b1c11763496f93fcc420d339e8a63d9b555563e547 WatchSource:0}: Error finding container 81be8b55f46d1c76e24e26b1c11763496f93fcc420d339e8a63d9b555563e547: Status 404 returned error can't find the container with id 81be8b55f46d1c76e24e26b1c11763496f93fcc420d339e8a63d9b555563e547
Oct 30 21:32:26.832847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.832823    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"]
Oct 30 21:32:26.900328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.900291    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g" event={"ID":"2a701134-7e5f-4b2b-81f5-6f1d1925a863","Type":"ContainerStarted","Data":"8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb"}
Oct 30 21:32:26.900513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.900362    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g" event={"ID":"2a701134-7e5f-4b2b-81f5-6f1d1925a863","Type":"ContainerStarted","Data":"689bdcb00ce0f18a27a719fefcb72cf447da6d0aa85e0faab9f22962f9fe0992"}
Oct 30 21:32:26.900513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.900454    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:26.901884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.901855    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5" event={"ID":"d73bf3bf-0b3d-417a-bf59-b86906557bad","Type":"ContainerStarted","Data":"81be8b55f46d1c76e24e26b1c11763496f93fcc420d339e8a63d9b555563e547"}
Oct 30 21:32:26.920392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:26.920338    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g" podStartSLOduration=1.920319982 podStartE2EDuration="1.920319982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:26.917519037 +0000 UTC m=+1855.267246431" watchObservedRunningTime="2026-10-30 21:32:26.920319982 +0000 UTC m=+1855.270047381"
Oct 30 21:32:27.710332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:27.710304    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:32:27.920717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:27.920644    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5" event={"ID":"d73bf3bf-0b3d-417a-bf59-b86906557bad","Type":"ContainerStarted","Data":"ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e"}
Oct 30 21:32:27.920717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:27.920717    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:27.947545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:27.947487    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5" podStartSLOduration=1.947468004 podStartE2EDuration="1.947468004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:27.940560619 +0000 UTC m=+1856.290288016" watchObservedRunningTime="2026-10-30 21:32:27.947468004 +0000 UTC m=+1856.297195400"
Oct 30 21:32:30.576239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.575773    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"]
Oct 30 21:32:30.586145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.586121    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.591433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.591410    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"]
Oct 30 21:32:30.600035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.600008    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7b5a7c-openshift-default-dockercfg-ftslg\""
Oct 30 21:32:30.628247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628203    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-credential-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.628454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628317    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e4e393-82cf-44c5-9247-321be52321a3-istio-podinfo\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.628454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628374    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e4e393-82cf-44c5-9247-321be52321a3-istiod-ca-cert\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.628580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628460    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-data\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.628580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628532    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-envoy\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.628716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628607    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmg94\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-kube-api-access-kmg94\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.628796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628781    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.628842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628816    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-certs\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.628842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.628836    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-istio-token\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730172    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730215    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-certs\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730234    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-istio-token\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730340    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-credential-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730380    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e4e393-82cf-44c5-9247-321be52321a3-istio-podinfo\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730435    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e4e393-82cf-44c5-9247-321be52321a3-istiod-ca-cert\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730470    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-data\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730541    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-envoy\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730597    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmg94\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-kube-api-access-kmg94\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730757    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-certs\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730865    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.730996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.730943    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-data\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.731120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.731049    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-credential-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.731358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.731333    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e4e393-82cf-44c5-9247-321be52321a3-istiod-ca-cert\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.733024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.733000    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-envoy\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.733414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.733391    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e4e393-82cf-44c5-9247-321be52321a3-istio-podinfo\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.741898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.741870    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmg94\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-kube-api-access-kmg94\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.742078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.742058    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-istio-token\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:30.915705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:30.911655    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:31.404394 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:31.404362    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e4e393_82cf_44c5_9247_321be52321a3.slice/crio-1f48cc4607dca5c509c7bb3167e212a8210ac987b81d63fed27eefa04ad54aab WatchSource:0}: Error finding container 1f48cc4607dca5c509c7bb3167e212a8210ac987b81d63fed27eefa04ad54aab: Status 404 returned error can't find the container with id 1f48cc4607dca5c509c7bb3167e212a8210ac987b81d63fed27eefa04ad54aab
Oct 30 21:32:31.407174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:31.407082    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"]
Oct 30 21:32:31.409843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:31.409793    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:31.409959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:31.409879    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:31.409959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:31.409932    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:31.412856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:31.412835    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"]
Oct 30 21:32:31.413038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:31.413017    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy" containerID="cri-o://7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb" gracePeriod=30
Oct 30 21:32:31.858790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:31.851847    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"]
Oct 30 21:32:31.858790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:31.852079    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy" containerID="cri-o://76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32" gracePeriod=30
Oct 30 21:32:32.056685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:32.056632    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" event={"ID":"a9e4e393-82cf-44c5-9247-321be52321a3","Type":"ContainerStarted","Data":"272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0"}
Oct 30 21:32:32.056910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:32.056720    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" event={"ID":"a9e4e393-82cf-44c5-9247-321be52321a3","Type":"ContainerStarted","Data":"1f48cc4607dca5c509c7bb3167e212a8210ac987b81d63fed27eefa04ad54aab"}
Oct 30 21:32:32.079558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:32.079502    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podStartSLOduration=2.079485857 podStartE2EDuration="2.079485857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:32.07618673 +0000 UTC m=+1860.425914126" watchObservedRunningTime="2026-10-30 21:32:32.079485857 +0000 UTC m=+1860.429213252"
Oct 30 21:32:32.912167 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:32.912133    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:32.918056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:32.918030    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:33.067402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.067374    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:33.068344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.068318    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:32:33.102823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.102790    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"]
Oct 30 21:32:33.108002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.107969    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.122998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.122956    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"]
Oct 30 21:32:33.123329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.123305    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-27804d-openshift-default-dockercfg-zjgbk\""
Oct 30 21:32:33.203223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.203138    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-credential-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.203389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.203339    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-istio-token\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.203530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.203514    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-certs\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.203603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.203563    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-envoy\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.203603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.203582    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592f635-0310-4fae-ad00-dec841dec9c7-istiod-ca-cert\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.203728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.203664    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592f635-0310-4fae-ad00-dec841dec9c7-istio-podinfo\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.203782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.203757    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-data\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.203851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.203821    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.204054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.204026    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76k4q\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-kube-api-access-76k4q\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305117    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-credential-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305154    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-istio-token\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305213    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-certs\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305255    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-envoy\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305283    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592f635-0310-4fae-ad00-dec841dec9c7-istiod-ca-cert\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305317    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592f635-0310-4fae-ad00-dec841dec9c7-istio-podinfo\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305378    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-data\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305456    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305629    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76k4q\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-kube-api-access-76k4q\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305654    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-credential-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305757    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-data\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.305975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.305933    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.306153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.306133    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592f635-0310-4fae-ad00-dec841dec9c7-istiod-ca-cert\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.306211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.306158    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-certs\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.307713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.307696    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-envoy\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.308200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.308182    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592f635-0310-4fae-ad00-dec841dec9c7-istio-podinfo\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.313252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.313230    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-istio-token\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.313366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.313355    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76k4q\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-kube-api-access-76k4q\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.463777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.462166    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:33.727911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.727835    4633 scope.go:122] "RemoveContainer" containerID="3dd2779997b8ca9f640e0d5411cda304b0a762f251a51e06c9f7acb2379ac053"
Oct 30 21:32:33.781912 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.781888    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"]
Oct 30 21:32:33.784217 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:33.784187    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6592f635_0310_4fae_ad00_dec841dec9c7.slice/crio-8fc000e4d7831c6506fc55fc4a17cee1f24285b2217abacd2a76f9bb7c0c841b WatchSource:0}: Error finding container 8fc000e4d7831c6506fc55fc4a17cee1f24285b2217abacd2a76f9bb7c0c841b: Status 404 returned error can't find the container with id 8fc000e4d7831c6506fc55fc4a17cee1f24285b2217abacd2a76f9bb7c0c841b
Oct 30 21:32:33.791047 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.791013    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:33.791155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.791094    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:33.791155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.791131    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:33.799658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.799610    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp" podUID="cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" containerName="main" containerID="cri-o://4cf4f9bc163b25fa8b8282016a91eeca56f2b1e632bc95556f70e717d7b07c51" gracePeriod=45
Oct 30 21:32:33.943805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:33.943753    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp" podUID="cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.196:8000/ready\": dial tcp 10.133.0.196:8000: connect: connection refused"
Oct 30 21:32:34.095048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.095004    4633 generic.go:356] "Generic (PLEG): container finished" podID="cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" containerID="4cf4f9bc163b25fa8b8282016a91eeca56f2b1e632bc95556f70e717d7b07c51" exitCode=0
Oct 30 21:32:34.095454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.095268    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp" event={"ID":"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b","Type":"ContainerDied","Data":"4cf4f9bc163b25fa8b8282016a91eeca56f2b1e632bc95556f70e717d7b07c51"}
Oct 30 21:32:34.098551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.098080    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" event={"ID":"6592f635-0310-4fae-ad00-dec841dec9c7","Type":"ContainerStarted","Data":"7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2"}
Oct 30 21:32:34.098551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.098123    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" event={"ID":"6592f635-0310-4fae-ad00-dec841dec9c7","Type":"ContainerStarted","Data":"8fc000e4d7831c6506fc55fc4a17cee1f24285b2217abacd2a76f9bb7c0c841b"}
Oct 30 21:32:34.130919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.130873    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podStartSLOduration=1.130858807 podStartE2EDuration="1.130858807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:34.123989715 +0000 UTC m=+1862.473717112" watchObservedRunningTime="2026-10-30 21:32:34.130858807 +0000 UTC m=+1862.480586273"
Oct 30 21:32:34.163416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.163386    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:32:34.325874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.325844    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-model-cache\") pod \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") "
Oct 30 21:32:34.326025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.325997    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tmp-dir\") pod \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") "
Oct 30 21:32:34.326091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.326034    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-dshm\") pod \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") "
Oct 30 21:32:34.326091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.326077    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-model-cache" pod "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" (UID: "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.326208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.326098    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-home\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-home\") pod \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") "
Oct 30 21:32:34.326208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.326203    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tls-certs\") pod \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") "
Oct 30 21:32:34.326322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.326297    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tmp-dir" pod "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" (UID: "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.326447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.326367    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-kube-api-access-zxxq8\" (UniqueName: \"kubernetes.io/projected/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-kube-api-access-zxxq8\") pod \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\" (UID: \"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b\") "
Oct 30 21:32:34.327116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.326601    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-home" pod "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" (UID: "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.327116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.327042    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.327116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.327065    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.327116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.327079    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.330459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.330414    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-dshm" pod "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" (UID: "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:34.330658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.330573    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-kube-api-access-zxxq8" pod "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" (UID: "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b"). InnerVolumeSpecName "kube-api-access-zxxq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:34.331084 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.331058    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tls-certs" pod "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" (UID: "cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:34.429179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.429137    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.429179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.429179    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.429398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.429196    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxq8\" (UniqueName: \"kubernetes.io/projected/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b-kube-api-access-zxxq8\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:34.463684 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.463637    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:34.465623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.465594    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 21:32:34.465832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:34.465648    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 21:32:35.108876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.108840    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"
Oct 30 21:32:35.109370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.108870    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp" event={"ID":"cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b","Type":"ContainerDied","Data":"e526b18c0f3ed894ae7909a249399f0d81cb509a839b5b8a154879ceaa3b6579"}
Oct 30 21:32:35.109370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.108924    4633 scope.go:122] "RemoveContainer" containerID="4cf4f9bc163b25fa8b8282016a91eeca56f2b1e632bc95556f70e717d7b07c51"
Oct 30 21:32:35.144086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.144048    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"]
Oct 30 21:32:35.151647 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.151618    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-904993/auth-test-model-904993-kserve-54f98948c7-hwhxp"]
Oct 30 21:32:35.463513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.463475    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 21:32:35.463767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.463541    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 21:32:35.754719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.754618    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"]
Oct 30 21:32:35.755713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.755689    4633 cpu_manager.go:414] "containerMap: removing container" podUID="cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" containerName="main"
Oct 30 21:32:35.755713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.755712    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" containerName="main"
Oct 30 21:32:35.755954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.755940    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" containerName="main"
Oct 30 21:32:35.761237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.761215    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:35.772611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.772585    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7b5a7c\""
Oct 30 21:32:35.772885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.772717    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7b5a7c-dockercfg-4qt7n\""
Oct 30 21:32:35.774569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.774543    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"]
Oct 30 21:32:35.796227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.796194    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b" path="/var/lib/kubelet/pods/cfc9d4ba-db00-477a-92a3-f0ed1a5c2e5b/volumes"
Oct 30 21:32:35.864061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.864023    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"]
Oct 30 21:32:35.865390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.865357    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns\" (UID: \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:35.865764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.865741    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txrn\" (UniqueName: \"kubernetes.io/projected/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-kube-api-access-4txrn\") pod \"payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns\" (UID: \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:35.870185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.870164    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:35.892845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.892815    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"]
Oct 30 21:32:35.993694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.985244    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0ea1ea-openshift-default-dockercfg-9tksn\""
Oct 30 21:32:36.035754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:35.995001    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.045965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.045930    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.046273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.046254    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txrn\" (UniqueName: \"kubernetes.io/projected/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-kube-api-access-4txrn\") pod \"payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns\" (UID: \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:36.061876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.061843    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.062223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.062200    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ef3b75-43f3-44ea-a266-7184d31c4331-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.062481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.062463    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.062738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.062720    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns\" (UID: \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:36.108939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.108904    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.116857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.116827    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.142038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.142000    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-kube-api-access-b2h9j\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.142411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.142381    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.167695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.098320    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns\" (UID: \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:36.257711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.199446    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txrn\" (UniqueName: \"kubernetes.io/projected/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-kube-api-access-4txrn\") pod \"payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns\" (UID: \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282061    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-kube-api-access-b2h9j\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282161    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282300    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282327    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282422    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282465    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ef3b75-43f3-44ea-a266-7184d31c4331-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282555    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282662    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.282736    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.283130    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.284800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.283479    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.286331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.286260    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.302729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.302051    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.345703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.334467    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ef3b75-43f3-44ea-a266-7184d31c4331-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.387949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.387627    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:36.434805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.428482    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-kube-api-access-b2h9j\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.464442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.463078    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.468292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.464937    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 21:32:36.468292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.464978    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 21:32:36.468292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.465979    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.468292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.466501    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.496164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.496134    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:36.763130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.762662    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"]
Oct 30 21:32:36.763130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.762920    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy" containerID="cri-o://8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931" gracePeriod=30
Oct 30 21:32:36.775566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.775095    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"]
Oct 30 21:32:36.775566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.775279    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy" containerID="cri-o://0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815" gracePeriod=30
Oct 30 21:32:36.782922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.782892    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"]
Oct 30 21:32:36.783197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.783169    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy" containerID="cri-o://c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f" gracePeriod=30
Oct 30 21:32:36.850268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.850237    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:32:36.900481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.900449    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"]
Oct 30 21:32:36.902184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902161    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-data\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902245    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-envoy\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902294    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7911991f-b0ac-4181-ac75-1b25bacca07f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7911991f-b0ac-4181-ac75-1b25bacca07f-istiod-ca-cert\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902322    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-kube-api-access-thgpt\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-kube-api-access-thgpt\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902417    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-token\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-token\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902470 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902458    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-credential-socket\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902497    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-podinfo\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902582    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-certs\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902690    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-socket\") pod \"7911991f-b0ac-4181-ac75-1b25bacca07f\" (UID: \"7911991f-b0ac-4181-ac75-1b25bacca07f\") "
Oct 30 21:32:36.902907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.902848    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-data" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:36.903094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.903068    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-certs" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:36.903094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.903077    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-socket" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:36.903280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.903205    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-credential-socket" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:36.903503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.903451    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:36.903503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.903472    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:36.903503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.903481    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:36.903503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.903495    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:36.903938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.903582    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7911991f-b0ac-4181-ac75-1b25bacca07f-istiod-ca-cert" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:36.905340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.905318    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-envoy" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:36.905612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.905589    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-podinfo" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:36.905879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.905851    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-kube-api-access-thgpt" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "kube-api-access-thgpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:36.906027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.906007    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-token" pod "7911991f-b0ac-4181-ac75-1b25bacca07f" (UID: "7911991f-b0ac-4181-ac75-1b25bacca07f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:36.971921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.971885    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:36.972016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.971968    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:36.972016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.972005    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:36.989764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:36.989730    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"]
Oct 30 21:32:36.989764 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:36.989756    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ce2b2c_c62b_474d_8af8_4350bb1ddee7.slice/crio-afcafee578fb842d73cbadca9cb5d4ebd4ad83685743a6639c31e5a60ea28a40 WatchSource:0}: Error finding container afcafee578fb842d73cbadca9cb5d4ebd4ad83685743a6639c31e5a60ea28a40: Status 404 returned error can't find the container with id afcafee578fb842d73cbadca9cb5d4ebd4ad83685743a6639c31e5a60ea28a40
Oct 30 21:32:37.037693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.011035    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.083937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.083813    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7911991f-b0ac-4181-ac75-1b25bacca07f-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.084389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.083844    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thgpt\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-kube-api-access-thgpt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.084627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.084567    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.084627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.084593    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7911991f-b0ac-4181-ac75-1b25bacca07f-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.356432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.356411    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:32:37.465929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.464933    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 21:32:37.465929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.465044    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 21:32:37.478417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.477940    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" event={"ID":"54ef3b75-43f3-44ea-a266-7184d31c4331","Type":"ContainerStarted","Data":"cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649"}
Oct 30 21:32:37.478417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.478001    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" event={"ID":"54ef3b75-43f3-44ea-a266-7184d31c4331","Type":"ContainerStarted","Data":"d140c3012044ba804d4f65909cf9a0769a6cf04aac109e0c456c51dbad2263b4"}
Oct 30 21:32:37.480291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.480264    4633 generic.go:356] "Generic (PLEG): container finished" podID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerID="7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb" exitCode=0
Oct 30 21:32:37.480442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.480309    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"
Oct 30 21:32:37.480442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.480352    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" event={"ID":"7911991f-b0ac-4181-ac75-1b25bacca07f","Type":"ContainerDied","Data":"7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb"}
Oct 30 21:32:37.480442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.480387    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q" event={"ID":"7911991f-b0ac-4181-ac75-1b25bacca07f","Type":"ContainerDied","Data":"41285996360a7c51f3015e699d77a24eea39e2f550c2a0ccdb979b18f5631416"}
Oct 30 21:32:37.480442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.480409    4633 scope.go:122] "RemoveContainer" containerID="7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb"
Oct 30 21:32:37.486029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.485600    4633 generic.go:356] "Generic (PLEG): container finished" podID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerID="76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32" exitCode=0
Oct 30 21:32:37.486029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.485767    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" event={"ID":"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946","Type":"ContainerDied","Data":"76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32"}
Oct 30 21:32:37.486029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.485812    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn" event={"ID":"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946","Type":"ContainerDied","Data":"ef0c6721ecc9cc50f0b305f7d161669897e6520414af6895eb5274ee1e938675"}
Oct 30 21:32:37.486029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.485886    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"
Oct 30 21:32:37.488714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.488475    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns" event={"ID":"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7","Type":"ContainerStarted","Data":"3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459"}
Oct 30 21:32:37.488714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.488520    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns" event={"ID":"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7","Type":"ContainerStarted","Data":"afcafee578fb842d73cbadca9cb5d4ebd4ad83685743a6639c31e5a60ea28a40"}
Oct 30 21:32:37.489694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.489616    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:37.514967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.506638    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515180    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-token\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-token\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515259    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-envoy\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515308    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istiod-ca-cert\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515339    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-podinfo\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515369    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-credential-socket\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515407    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-socket\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515457    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-data\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515491    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-certs\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.521691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.515764    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-kube-api-access-qt7kx\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-kube-api-access-qt7kx\") pod \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\" (UID: \"8bb6133f-bf81-4b1d-9fb2-1f447b5c4946\") "
Oct 30 21:32:37.586694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.578297    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-certs" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.586694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.578865    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-credential-socket" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.586694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.579088    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-socket" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.595478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.595295    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istiod-ca-cert" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:37.607927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.607825    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-token" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:37.645173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.618745    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-kube-api-access-qt7kx" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "kube-api-access-qt7kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:37.645173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.619048    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns" podStartSLOduration=2.619030559 podStartE2EDuration="2.619030559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:37.61542759 +0000 UTC m=+1865.965154997" watchObservedRunningTime="2026-10-30 21:32:37.619030559 +0000 UTC m=+1865.968757956"
Oct 30 21:32:37.645173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.619265    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podStartSLOduration=2.619257555 podStartE2EDuration="2.619257555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:37.595497151 +0000 UTC m=+1865.945224549" watchObservedRunningTime="2026-10-30 21:32:37.619257555 +0000 UTC m=+1865.968984952"
Oct 30 21:32:37.667015 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.666981    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-data" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.667413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.667360    4633 scope.go:122] "RemoveContainer" containerID="7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb"
Oct 30 21:32:37.667584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.667560    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-podinfo" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:37.667720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.667654    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-envoy" pod "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" (UID: "8bb6133f-bf81-4b1d-9fb2-1f447b5c4946"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.668373 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:37.668338    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb\": container with ID starting with 7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb not found: ID does not exist" containerID="7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb"
Oct 30 21:32:37.668482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.668384    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb"} err="failed to get container status \"7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb\": rpc error: code = NotFound desc = could not find container \"7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb\": container with ID starting with 7a7fe609ef9db52d965830c8d305eb3133eb7a47c49cf14dc3aa39a4a1c180bb not found: ID does not exist"
Oct 30 21:32:37.668482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.668409    4633 scope.go:122] "RemoveContainer" containerID="76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32"
Oct 30 21:32:37.695579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.695529    4633 scope.go:122] "RemoveContainer" containerID="76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32"
Oct 30 21:32:37.696053 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:37.696013    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32\": container with ID starting with 76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32 not found: ID does not exist" containerID="76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32"
Oct 30 21:32:37.696161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.696054    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32"} err="failed to get container status \"76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32\": rpc error: code = NotFound desc = could not find container \"76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32\": container with ID starting with 76b9c73e73ef52eeaaa576219ff0677488402e6a635aa75c75822207a3e73d32 not found: ID does not exist"
Oct 30 21:32:37.696704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.696646    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"]
Oct 30 21:32:37.702051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.702015    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-wlw8q"]
Oct 30 21:32:37.735762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.735720    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt7kx\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-kube-api-access-qt7kx\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.736007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.735986    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.736130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.736117    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.736226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.736214    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.736358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.736346    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.736476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.736462    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.736573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.736562    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.736657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.736646    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.736765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.736753    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.861284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.861177    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" path="/var/lib/kubelet/pods/7911991f-b0ac-4181-ac75-1b25bacca07f/volumes"
Oct 30 21:32:37.884572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.884537    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"]
Oct 30 21:32:37.889246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.889222    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-22mcn"]
Oct 30 21:32:37.927927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:37.923771    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:32:38.463533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:38.463438    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 21:32:38.463979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:38.463522    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 21:32:38.502748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:38.502721    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:38.944171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:38.944144    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:32:39.198593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.198493    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"]
Oct 30 21:32:39.199386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.199366    4633 cpu_manager.go:414] "containerMap: removing container" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy"
Oct 30 21:32:39.199386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.199384    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy"
Oct 30 21:32:39.199654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.199417    4633 cpu_manager.go:414] "containerMap: removing container" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy"
Oct 30 21:32:39.199654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.199423    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy"
Oct 30 21:32:39.199654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.199535    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7911991f-b0ac-4181-ac75-1b25bacca07f" containerName="istio-proxy"
Oct 30 21:32:39.199654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.199548    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" containerName="istio-proxy"
Oct 30 21:32:39.206185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.206164    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.241998    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-1f1b14-openshift-default-dockercfg-tjlqm\""
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.243893    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-podinfo\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.243940    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-envoy\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.244004    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-certs\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.244137    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-token\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.244213    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-credential-socket\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.244284    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dae388c-07af-45f1-b4c5-57bc563b6a14-istiod-ca-cert\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.244318    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhhg\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-kube-api-access-pmhhg\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.244371    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-data\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.245097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.244408    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-socket\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.254731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.254460    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"]
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.348746    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-podinfo\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.366797    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-envoy\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.366892    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-certs\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.367125    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-token\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.367235    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-credential-socket\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.367332    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dae388c-07af-45f1-b4c5-57bc563b6a14-istiod-ca-cert\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.367378    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhhg\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-kube-api-access-pmhhg\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.367444    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-data\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.367692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.367492    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-socket\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.403694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.394408    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-credential-socket\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.403694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.396225    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-socket\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.403694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.396512    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-certs\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.403694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.397384    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dae388c-07af-45f1-b4c5-57bc563b6a14-istiod-ca-cert\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.403694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.399162    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-data\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.543746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.530006    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-podinfo\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.543746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.538804    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 21:32:39.543746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.538864    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 21:32:39.543746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.541976    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-envoy\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.566695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.558501    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhhg\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-kube-api-access-pmhhg\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.574748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.570354    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:39.587767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.578935    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:32:39.587767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.579344    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-token\") pod \"e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") " pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:39.796930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.796856    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb6133f-bf81-4b1d-9fb2-1f447b5c4946" path="/var/lib/kubelet/pods/8bb6133f-bf81-4b1d-9fb2-1f447b5c4946/volumes"
Oct 30 21:32:39.855969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:39.855941    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:40.353101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.344624    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"]
Oct 30 21:32:40.364356 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:40.364325    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dae388c_07af_45f1_b4c5_57bc563b6a14.slice/crio-35c44e20346bdb385d0f860ca52188c48ee1c4b2feb2ef4ef3fd5974fdd1a71b WatchSource:0}: Error finding container 35c44e20346bdb385d0f860ca52188c48ee1c4b2feb2ef4ef3fd5974fdd1a71b: Status 404 returned error can't find the container with id 35c44e20346bdb385d0f860ca52188c48ee1c4b2feb2ef4ef3fd5974fdd1a71b
Oct 30 21:32:40.380289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.380217    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:40.380412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.380318    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:40.380412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.380372    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:32:40.464687 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.464645    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 21:32:40.464830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.464722    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 21:32:40.587953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.587859    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f" event={"ID":"1dae388c-07af-45f1-b4c5-57bc563b6a14","Type":"ContainerStarted","Data":"eb8d2bab89e0f6661a93566562fb2998b5a5befa61f352857c3e17385b6e0beb"}
Oct 30 21:32:40.587953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.587924    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f" event={"ID":"1dae388c-07af-45f1-b4c5-57bc563b6a14","Type":"ContainerStarted","Data":"35c44e20346bdb385d0f860ca52188c48ee1c4b2feb2ef4ef3fd5974fdd1a71b"}
Oct 30 21:32:40.645570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.645440    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f" podStartSLOduration=1.645420922 podStartE2EDuration="1.645420922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:40.61110099 +0000 UTC m=+1868.960828389" watchObservedRunningTime="2026-10-30 21:32:40.645420922 +0000 UTC m=+1868.995148319"
Oct 30 21:32:40.865697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:40.857639    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:41.529701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:41.527868    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Oct 30 21:32:41.529701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:41.527949    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Oct 30 21:32:41.867189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:41.867138    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:42.231978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.231597    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:32:42.239725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.238150    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:32:42.311724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.311165    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:32:42.353762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.345128    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-kube-api-access-wvgv2\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-kube-api-access-wvgv2\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.353762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.345240    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b81a30-63dd-45c4-b183-75a67e3f37b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b81a30-63dd-45c4-b183-75a67e3f37b6-istiod-ca-cert\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.353762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.345363    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-certs\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.353762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.345478    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-envoy\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.353762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.345541    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-podinfo\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.353762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.345632    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-credential-socket\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.353762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.345688    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-token\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-token\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.356975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.345769    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-data\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.356975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.356797    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-socket\") pod \"16b81a30-63dd-45c4-b183-75a67e3f37b6\" (UID: \"16b81a30-63dd-45c4-b183-75a67e3f37b6\") "
Oct 30 21:32:42.359974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.359712    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-certs" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.360101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.360011    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b81a30-63dd-45c4-b183-75a67e3f37b6-istiod-ca-cert" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:42.360169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.360101    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-data" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.360706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.360684    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-credential-socket" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.375336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.361133    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-socket" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.375336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.371646    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-kube-api-access-wvgv2" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "kube-api-access-wvgv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:42.376863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.376830    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-token" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:42.385452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.385376    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-envoy" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.388038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.387643    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:32:42.389347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.389303    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-podinfo" pod "16b81a30-63dd-45c4-b183-75a67e3f37b6" (UID: "16b81a30-63dd-45c4-b183-75a67e3f37b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:42.460113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.460062    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-envoy\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.460836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.460808    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-podinfo\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.461106 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.461085    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-podinfo\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.461265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.461250    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-token\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-token\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.461385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.461369    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-socket\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.461605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.461523    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istiod-ca-cert\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.461767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.461750    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-token\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-token\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.464823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.464792    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-data\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.464944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.464843    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-kube-api-access-5lsd4\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-kube-api-access-5lsd4\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.464944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.464908    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istiod-ca-cert\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.465072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.464988    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-credential-socket\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.465072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.465034    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-data\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.465188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.465069    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-kube-api-access-6z8zb\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-kube-api-access-6z8zb\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.465188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.465164    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-envoy\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.465355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.465292    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-socket\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.465416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.465371    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-certs\") pod \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\" (UID: \"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0\") "
Oct 30 21:32:42.465416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.465400    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-certs\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.465526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.465439    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-credential-socket\") pod \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\" (UID: \"d024cbfe-5e99-4237-bece-8ddfba6e3d3a\") "
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466362    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466388    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466403    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466417    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466431    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466443    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466456    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvgv2\" (UniqueName: \"kubernetes.io/projected/16b81a30-63dd-45c4-b183-75a67e3f37b6-kube-api-access-wvgv2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466472    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b81a30-63dd-45c4-b183-75a67e3f37b6-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.466485    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b81a30-63dd-45c4-b183-75a67e3f37b6-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.472659    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-credential-socket" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.474966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.472918    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-data" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.475911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.475882    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-credential-socket" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.476223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.476190    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istiod-ca-cert" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:42.476468 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.476440    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-data" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.477053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.477027    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-socket" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.477409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.477376    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-socket" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.480399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.480113    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istiod-ca-cert" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:42.480743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.480548    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-certs" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.480933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.480900    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-certs" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.481148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.481128    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-envoy" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.485575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.485551    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-podinfo" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:42.488006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.487983    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-kube-api-access-5lsd4" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "kube-api-access-5lsd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:42.491241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.491215    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-kube-api-access-6z8zb" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "kube-api-access-6z8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:42.491686 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.491635    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-podinfo" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:42.491777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.491707    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-token" pod "d024cbfe-5e99-4237-bece-8ddfba6e3d3a" (UID: "d024cbfe-5e99-4237-bece-8ddfba6e3d3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:42.492879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.492854    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-envoy" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:42.493939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.493911    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-token" pod "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" (UID: "ed0932eb-ba19-4b6f-9b2b-324572cd2aa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590238    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590278    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590292    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590305    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590320    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590332    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590345    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lsd4\" (UniqueName: \"kubernetes.io/projected/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-kube-api-access-5lsd4\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590360    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590374    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590388    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590403    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8zb\" (UniqueName: \"kubernetes.io/projected/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-kube-api-access-6z8zb\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590421    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590435    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590450    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590464    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590478    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590490    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.603740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.590501    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d024cbfe-5e99-4237-bece-8ddfba6e3d3a-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:42.755922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.738318    4633 generic.go:356] "Generic (PLEG): container finished" podID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerID="8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931" exitCode=0
Oct 30 21:32:42.756512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.755749    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" event={"ID":"16b81a30-63dd-45c4-b183-75a67e3f37b6","Type":"ContainerDied","Data":"8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931"}
Oct 30 21:32:42.756856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.756833    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd" event={"ID":"16b81a30-63dd-45c4-b183-75a67e3f37b6","Type":"ContainerDied","Data":"9ffc36ac82726e766b77836afebbf4b4a5b7cbab99f0279e0cf500f2ab3b95bb"}
Oct 30 21:32:42.756996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.756979    4633 scope.go:122] "RemoveContainer" containerID="8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931"
Oct 30 21:32:42.757197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.755893    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"
Oct 30 21:32:42.779485 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.779455    4633 generic.go:356] "Generic (PLEG): container finished" podID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerID="c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f" exitCode=0
Oct 30 21:32:42.794230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.794183    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" event={"ID":"d024cbfe-5e99-4237-bece-8ddfba6e3d3a","Type":"ContainerDied","Data":"c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f"}
Oct 30 21:32:42.794492 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.794467    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8" event={"ID":"d024cbfe-5e99-4237-bece-8ddfba6e3d3a","Type":"ContainerDied","Data":"28bf59232d6b3705d050e05413b8ee276ddcb258f2dda20d30a0cd141ff89a3e"}
Oct 30 21:32:42.794846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.794825    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"
Oct 30 21:32:42.814699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.806312    4633 generic.go:356] "Generic (PLEG): container finished" podID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerID="0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815" exitCode=0
Oct 30 21:32:42.814699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.813350    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" event={"ID":"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0","Type":"ContainerDied","Data":"0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815"}
Oct 30 21:32:42.814699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.813414    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw" event={"ID":"ed0932eb-ba19-4b6f-9b2b-324572cd2aa0","Type":"ContainerDied","Data":"e81af8b5aff4c7f9c1c6181c3b0ac49cfc1ab97df8660c95ceb2e8d6446d83fb"}
Oct 30 21:32:42.825118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.825075    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:42.825432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.825407    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"
Oct 30 21:32:42.863097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:42.853234    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:43.219721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.219015    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:43.222546 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.220969    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:43.223574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.223485    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:32:43.285098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.266017    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"]
Oct 30 21:32:43.285098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.283380    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-mpnvw"]
Oct 30 21:32:43.347558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.347466    4633 scope.go:122] "RemoveContainer" containerID="8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931"
Oct 30 21:32:43.399702 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:43.399530    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931\": container with ID starting with 8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931 not found: ID does not exist" containerID="8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931"
Oct 30 21:32:43.399702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.399580    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931"} err="failed to get container status \"8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931\": rpc error: code = NotFound desc = could not find container \"8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931\": container with ID starting with 8eb6879c506557f81bc4bdbd700d92261a3aa30c441a2c5bc83f08cddebb9931 not found: ID does not exist"
Oct 30 21:32:43.399702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.399611    4633 scope.go:122] "RemoveContainer" containerID="c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f"
Oct 30 21:32:43.996692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.996645    4633 scope.go:122] "RemoveContainer" containerID="c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f"
Oct 30 21:32:43.997642 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:43.997619    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f\": container with ID starting with c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f not found: ID does not exist" containerID="c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f"
Oct 30 21:32:43.997821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.997793    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f"} err="failed to get container status \"c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f\": rpc error: code = NotFound desc = could not find container \"c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f\": container with ID starting with c8283c7034697d0dfe87073923e27508fd42359cd1502a152b0cb6ab08d7644f not found: ID does not exist"
Oct 30 21:32:43.997927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:43.997917    4633 scope.go:122] "RemoveContainer" containerID="0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815"
Oct 30 21:32:44.351700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.351209    4633 scope.go:122] "RemoveContainer" containerID="0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815"
Oct 30 21:32:44.408975 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:44.408089    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815\": container with ID starting with 0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815 not found: ID does not exist" containerID="0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815"
Oct 30 21:32:44.408975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.408132    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815"} err="failed to get container status \"0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815\": rpc error: code = NotFound desc = could not find container \"0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815\": container with ID starting with 0ea517aaa7b637255fb95224bfc9ae00fd661dad5d602bc6558fdc2807b17815 not found: ID does not exist"
Oct 30 21:32:44.409630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.409261    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" path="/var/lib/kubelet/pods/ed0932eb-ba19-4b6f-9b2b-324572cd2aa0/volumes"
Oct 30 21:32:44.410498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.409817    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"]
Oct 30 21:32:44.410498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.409857    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-2chjd"]
Oct 30 21:32:44.410498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.409879    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"]
Oct 30 21:32:44.410498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.409894    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-shkj8"]
Oct 30 21:32:44.410498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.409910    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"]
Oct 30 21:32:44.411467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.411446    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy"
Oct 30 21:32:44.411579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.411568    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy"
Oct 30 21:32:44.411890 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.411877    4633 cpu_manager.go:414] "containerMap: removing container" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy"
Oct 30 21:32:44.411989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.411978    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy"
Oct 30 21:32:44.412081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.412072    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy"
Oct 30 21:32:44.412153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.412144    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy"
Oct 30 21:32:44.412440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.412427    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" containerName="istio-proxy"
Oct 30 21:32:44.412523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.412514    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" containerName="istio-proxy"
Oct 30 21:32:44.412614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.412605    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0932eb-ba19-4b6f-9b2b-324572cd2aa0" containerName="istio-proxy"
Oct 30 21:32:44.458868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.458824    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"]
Oct 30 21:32:44.459246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.459226    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.463997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.463969    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-02ecd7-serving-cert\""
Oct 30 21:32:44.464315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.464291    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gv2\" (UniqueName: \"kubernetes.io/projected/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-kube-api-access-z8gv2\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.464906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.464883    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-metrics-certs\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.465080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.465061    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-openshift-service-ca\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.465399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.465378    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-maas-api-tls\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.567775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.567743    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gv2\" (UniqueName: \"kubernetes.io/projected/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-kube-api-access-z8gv2\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.567988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.567810    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-metrics-certs\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.567988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.567855    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-openshift-service-ca\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.567988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.567958    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-maas-api-tls\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.568855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.568831    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-openshift-service-ca\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.571107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.571089    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-maas-api-tls\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.571577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.571530    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-metrics-certs\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.582458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.582372    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gv2\" (UniqueName: \"kubernetes.io/projected/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-kube-api-access-z8gv2\") pod \"maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:44.802725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:44.797205    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:45.270337 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:45.270303    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd4f737_8cfd_4f31_b2ae_f5c40ff66f8a.slice/crio-956fc071608242ba7a70004238d136dfb88494323f75bbe4974663cb843b9e62 WatchSource:0}: Error finding container 956fc071608242ba7a70004238d136dfb88494323f75bbe4974663cb843b9e62: Status 404 returned error can't find the container with id 956fc071608242ba7a70004238d136dfb88494323f75bbe4974663cb843b9e62
Oct 30 21:32:45.271539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:45.271510    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"]
Oct 30 21:32:45.796759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:45.796725    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b81a30-63dd-45c4-b183-75a67e3f37b6" path="/var/lib/kubelet/pods/16b81a30-63dd-45c4-b183-75a67e3f37b6/volumes"
Oct 30 21:32:45.797620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:45.797591    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d024cbfe-5e99-4237-bece-8ddfba6e3d3a" path="/var/lib/kubelet/pods/d024cbfe-5e99-4237-bece-8ddfba6e3d3a/volumes"
Oct 30 21:32:46.096973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:46.096886    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn" event={"ID":"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a","Type":"ContainerStarted","Data":"4f496328c1c71355b23832e4bcf77b0d9180200a1a481e135b9d1a4fe7746654"}
Oct 30 21:32:46.096973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:46.096936    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn" event={"ID":"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a","Type":"ContainerStarted","Data":"956fc071608242ba7a70004238d136dfb88494323f75bbe4974663cb843b9e62"}
Oct 30 21:32:46.097208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:46.096983    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:46.118054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:46.118005    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn" podStartSLOduration=1.696183405 podStartE2EDuration="2.117987801s" totalImagesPullingTime="421.804396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:46.115529326 +0000 UTC m=+1874.465256723" watchObservedRunningTime="2026-10-30 21:32:46.117987801 +0000 UTC m=+1874.467715199"
Oct 30 21:32:49.114693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.094122    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"]
Oct 30 21:32:49.122774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.117448    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"]
Oct 30 21:32:49.122774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.117564    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.136186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.136161    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1f1b14\"/\"openshift-service-ca.crt\""
Oct 30 21:32:49.138687 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.138625    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1f1b14\"/\"default-dockercfg-jttlv\""
Oct 30 21:32:49.138977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.138959    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1f1b14\"/\"kube-root-ca.crt\""
Oct 30 21:32:49.139220 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.139204    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1f1b14\"/\"noten-model-1f1b14-kserve-self-signed-certs\""
Oct 30 21:32:49.220895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.220854    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-dshm\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.221111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.220952    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tls-certs\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.221111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.221032    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-home\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.221111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.221056    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-model-cache\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.221289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.221134    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsznr\" (UniqueName: \"kubernetes.io/projected/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-kube-api-access-lsznr\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.221352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.221294    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tmp-dir\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.322780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.322743    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsznr\" (UniqueName: \"kubernetes.io/projected/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-kube-api-access-lsznr\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.322995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.322931    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tmp-dir\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.323079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.323048    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-dshm\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.323165 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.323146    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tls-certs\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.323256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.323245    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-home\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.323315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.323269    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-model-cache\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.323479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.323450    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tmp-dir\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.323714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.323690    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-model-cache\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.323854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.323836    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-home\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.326176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.326145    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-dshm\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.326399 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.326378    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tls-certs\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.336548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.336514    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsznr\" (UniqueName: \"kubernetes.io/projected/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-kube-api-access-lsznr\") pod \"noten-model-1f1b14-kserve-6ff44d869-qvdv8\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") " pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.448750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.445574    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:49.576506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.576473    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:32:49.843766 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:49.843723    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73b2ff52_b6e7_41ca_9bd7_67fb01a95c27.slice/crio-b5b823ed942e291d8388d073f24fe6742f67da79c3cefcc30b69ae64fd3a0d65 WatchSource:0}: Error finding container b5b823ed942e291d8388d073f24fe6742f67da79c3cefcc30b69ae64fd3a0d65: Status 404 returned error can't find the container with id b5b823ed942e291d8388d073f24fe6742f67da79c3cefcc30b69ae64fd3a0d65
Oct 30 21:32:49.846923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:49.846896    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"]
Oct 30 21:32:50.154456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:50.154421    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8" event={"ID":"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27","Type":"ContainerStarted","Data":"b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece"}
Oct 30 21:32:50.154917 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:50.154466    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8" event={"ID":"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27","Type":"ContainerStarted","Data":"b5b823ed942e291d8388d073f24fe6742f67da79c3cefcc30b69ae64fd3a0d65"}
Oct 30 21:32:50.176472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:50.176414    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8" podStartSLOduration=1.1764004510000001 podStartE2EDuration="1.176400451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:50.171047639 +0000 UTC m=+1878.520775036" watchObservedRunningTime="2026-10-30 21:32:50.176400451 +0000 UTC m=+1878.526127884"
Oct 30 21:32:50.447029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:50.446944    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:32:52.111522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:52.111492    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:32:53.389168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:53.389133    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"]
Oct 30 21:32:53.971186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:53.971129    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"]
Oct 30 21:32:53.986752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:53.986653    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f" podUID="1dae388c-07af-45f1-b4c5-57bc563b6a14" containerName="istio-proxy" containerID="cri-o://eb8d2bab89e0f6661a93566562fb2998b5a5befa61f352857c3e17385b6e0beb" gracePeriod=30
Oct 30 21:32:54.349079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.348983    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"]
Oct 30 21:32:54.354954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.354925    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.368707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.368634    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"]
Oct 30 21:32:54.370375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.369961    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-7b5a7c-serving-cert\""
Oct 30 21:32:54.439757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.439710    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dg86\" (UniqueName: \"kubernetes.io/projected/61147a40-6652-4743-ab47-9e9c49a4b880-kube-api-access-6dg86\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.440159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.439873    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.440159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.440073    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61147a40-6652-4743-ab47-9e9c49a4b880-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.440159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.440126    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-metrics-certs\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.541994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.541956    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-metrics-certs\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.542173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.542031    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dg86\" (UniqueName: \"kubernetes.io/projected/61147a40-6652-4743-ab47-9e9c49a4b880-kube-api-access-6dg86\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.542173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.542094    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.542327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.542309    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61147a40-6652-4743-ab47-9e9c49a4b880-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.542989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.542964    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61147a40-6652-4743-ab47-9e9c49a4b880-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.544645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.544621    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.544763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.544739    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-metrics-certs\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.553924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.553636    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dg86\" (UniqueName: \"kubernetes.io/projected/61147a40-6652-4743-ab47-9e9c49a4b880-kube-api-access-6dg86\") pod \"maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.678565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.678534    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:54.970870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:54.970838    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"]
Oct 30 21:32:54.971105 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:32:54.971085    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61147a40_6652_4743_ab47_9e9c49a4b880.slice/crio-a8c93a2cf8aeb368f9f56114fb196672d903bf7517c4677aafd6fa7d047eda42 WatchSource:0}: Error finding container a8c93a2cf8aeb368f9f56114fb196672d903bf7517c4677aafd6fa7d047eda42: Status 404 returned error can't find the container with id a8c93a2cf8aeb368f9f56114fb196672d903bf7517c4677aafd6fa7d047eda42
Oct 30 21:32:55.023770 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:32:55.023728    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:32:55.247755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:55.247508    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq" event={"ID":"61147a40-6652-4743-ab47-9e9c49a4b880","Type":"ContainerStarted","Data":"a8c93a2cf8aeb368f9f56114fb196672d903bf7517c4677aafd6fa7d047eda42"}
Oct 30 21:32:56.258368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:56.258333    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq" event={"ID":"61147a40-6652-4743-ab47-9e9c49a4b880","Type":"ContainerStarted","Data":"2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac"}
Oct 30 21:32:56.258974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:56.258484    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:32:56.277996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:56.277943    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq" podStartSLOduration=1.8214137479999999 podStartE2EDuration="2.277928732s" totalImagesPullingTime="456.514984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:56.275353339 +0000 UTC m=+1884.625080742" watchObservedRunningTime="2026-10-30 21:32:56.277928732 +0000 UTC m=+1884.627656127"
Oct 30 21:32:57.824048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:57.824003    4633 patch_prober.go:29] interesting pod/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:32:57.824929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:57.824872    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f" podUID="1dae388c-07af-45f1-b4c5-57bc563b6a14" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:32:59.289100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.289070    4633 generic.go:356] "Generic (PLEG): container finished" podID="1dae388c-07af-45f1-b4c5-57bc563b6a14" containerID="eb8d2bab89e0f6661a93566562fb2998b5a5befa61f352857c3e17385b6e0beb" exitCode=0
Oct 30 21:32:59.289541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.289148    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f" event={"ID":"1dae388c-07af-45f1-b4c5-57bc563b6a14","Type":"ContainerDied","Data":"eb8d2bab89e0f6661a93566562fb2998b5a5befa61f352857c3e17385b6e0beb"}
Oct 30 21:32:59.316337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.316315    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:32:59.433521 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.433488    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-podinfo\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.433743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.433572    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-socket\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.433743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.433716    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-envoy\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.433743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.433742    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-kube-api-access-pmhhg\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-kube-api-access-pmhhg\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.433913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.433774    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-data\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.433913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.433810    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-credential-socket\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.434025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.433920    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-token\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-token\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.434025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.433972    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-certs\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.434138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.434006    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-socket" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:59.434138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.434040    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dae388c-07af-45f1-b4c5-57bc563b6a14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dae388c-07af-45f1-b4c5-57bc563b6a14-istiod-ca-cert\") pod \"1dae388c-07af-45f1-b4c5-57bc563b6a14\" (UID: \"1dae388c-07af-45f1-b4c5-57bc563b6a14\") "
Oct 30 21:32:59.434138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.434104    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-data" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:59.434328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.434129    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-credential-socket" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:59.434413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.434388    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-certs" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:59.434484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.434409    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dae388c-07af-45f1-b4c5-57bc563b6a14-istiod-ca-cert" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:59.435299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.435270    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:59.435431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.435301    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:59.435431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.435317    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:59.435431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.435333    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:59.435431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.435346    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dae388c-07af-45f1-b4c5-57bc563b6a14-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:59.436039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.436016    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-podinfo" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:59.436203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.436181    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-kube-api-access-pmhhg" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "kube-api-access-pmhhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:59.436276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.436227    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-envoy" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:59.436497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.436483    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-token" pod "1dae388c-07af-45f1-b4c5-57bc563b6a14" (UID: "1dae388c-07af-45f1-b4c5-57bc563b6a14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:59.536970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.536884    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:59.536970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.536914    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhhg\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-kube-api-access-pmhhg\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:59.536970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.536928    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:59.536970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:32:59.536940    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dae388c-07af-45f1-b4c5-57bc563b6a14-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:00.299648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.299621    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"
Oct 30 21:33:00.300117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.299647    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f" event={"ID":"1dae388c-07af-45f1-b4c5-57bc563b6a14","Type":"ContainerDied","Data":"35c44e20346bdb385d0f860ca52188c48ee1c4b2feb2ef4ef3fd5974fdd1a71b"}
Oct 30 21:33:00.300117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.299785    4633 scope.go:122] "RemoveContainer" containerID="eb8d2bab89e0f6661a93566562fb2998b5a5befa61f352857c3e17385b6e0beb"
Oct 30 21:33:00.327197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.327170    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"]
Oct 30 21:33:00.337904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.337881    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-1f1b14-openshift-default-cddd49bbc-dcf2f"]
Oct 30 21:33:00.362373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.362344    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"]
Oct 30 21:33:00.363181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.363163    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1dae388c-07af-45f1-b4c5-57bc563b6a14" containerName="istio-proxy"
Oct 30 21:33:00.363264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.363182    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dae388c-07af-45f1-b4c5-57bc563b6a14" containerName="istio-proxy"
Oct 30 21:33:00.363321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.363296    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dae388c-07af-45f1-b4c5-57bc563b6a14" containerName="istio-proxy"
Oct 30 21:33:00.368648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.368623    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.382074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.374530    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0ea1ea-serving-cert\""
Oct 30 21:33:00.390153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.390125    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"]
Oct 30 21:33:00.557986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.554956    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.557986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.555035    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvpr6\" (UniqueName: \"kubernetes.io/projected/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-kube-api-access-rvpr6\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.557986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.555104    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-metrics-certs\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.557986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.555537    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.657564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.657528    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.657871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.657590    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvpr6\" (UniqueName: \"kubernetes.io/projected/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-kube-api-access-rvpr6\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.657871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.657637    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-metrics-certs\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.658023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.657999    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.658724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.658696    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.661056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.661028    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.662490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.662470    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-metrics-certs\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.667471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.667417    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvpr6\" (UniqueName: \"kubernetes.io/projected/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-kube-api-access-rvpr6\") pod \"maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.691308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.691274    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:00.993700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:00.992429    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"]
Oct 30 21:33:01.331789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:01.331687    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx" event={"ID":"8d36d0a1-89c6-4aa8-8e38-0d18da409c93","Type":"ContainerStarted","Data":"6dc831cbbbc84440e00a43ba77a5e98a98b2915763cb708bf359fd07992e37bb"}
Oct 30 21:33:01.795531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:01.795492    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dae388c-07af-45f1-b4c5-57bc563b6a14" path="/var/lib/kubelet/pods/1dae388c-07af-45f1-b4c5-57bc563b6a14/volumes"
Oct 30 21:33:02.294003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:02.293939    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:33:02.352457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:02.352422    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx" event={"ID":"8d36d0a1-89c6-4aa8-8e38-0d18da409c93","Type":"ContainerStarted","Data":"b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762"}
Oct 30 21:33:02.353190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:02.353155    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:02.377866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:02.377813    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx" podStartSLOduration=1.9598976110000002 podStartE2EDuration="2.377796265s" totalImagesPullingTime="417.898654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:02.370589576 +0000 UTC m=+1890.720317017" watchObservedRunningTime="2026-10-30 21:33:02.377796265 +0000 UTC m=+1890.727523661"
Oct 30 21:33:08.366995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.366961    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:33:08.438562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.438500    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8" podUID="73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" containerName="main" containerID="cri-o://b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece" gracePeriod=45
Oct 30 21:33:08.741332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.741303    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:33:08.775074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775043    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tls-certs\") pod \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") "
Oct 30 21:33:08.775302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775122    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-dshm\") pod \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") "
Oct 30 21:33:08.775302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775229    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-model-cache\") pod \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") "
Oct 30 21:33:08.775302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775295    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tmp-dir\") pod \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") "
Oct 30 21:33:08.775563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775349    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-kube-api-access-lsznr\" (UniqueName: \"kubernetes.io/projected/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-kube-api-access-lsznr\") pod \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") "
Oct 30 21:33:08.775563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775411    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-home\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-home\") pod \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\" (UID: \"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27\") "
Oct 30 21:33:08.775563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775540    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-model-cache" pod "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" (UID: "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:08.775563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775549    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tmp-dir" pod "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" (UID: "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:08.775881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.775858    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-home" pod "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" (UID: "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:08.776292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.776266    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:08.776419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.776296    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:08.776419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.776310    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:08.778272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.778241    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-dshm" pod "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" (UID: "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:08.778371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.778295    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-kube-api-access-lsznr" pod "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" (UID: "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27"). InnerVolumeSpecName "kube-api-access-lsznr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:08.778541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.778517    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tls-certs" pod "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" (UID: "73b2ff52-b6e7-41ca-9bd7-67fb01a95c27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:08.878235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.878192    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:08.878235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.878225    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:08.878235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:08.878234    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsznr\" (UniqueName: \"kubernetes.io/projected/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27-kube-api-access-lsznr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:09.448526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.448489    4633 generic.go:356] "Generic (PLEG): container finished" podID="73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" containerID="b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece" exitCode=0
Oct 30 21:33:09.449008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.448531    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"
Oct 30 21:33:09.449008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.448572    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8" event={"ID":"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27","Type":"ContainerDied","Data":"b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece"}
Oct 30 21:33:09.449008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.448615    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8" event={"ID":"73b2ff52-b6e7-41ca-9bd7-67fb01a95c27","Type":"ContainerDied","Data":"b5b823ed942e291d8388d073f24fe6742f67da79c3cefcc30b69ae64fd3a0d65"}
Oct 30 21:33:09.449008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.448633    4633 scope.go:122] "RemoveContainer" containerID="b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece"
Oct 30 21:33:09.461229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.461213    4633 scope.go:122] "RemoveContainer" containerID="b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece"
Oct 30 21:33:09.461499 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:09.461483    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece\": container with ID starting with b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece not found: ID does not exist" containerID="b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece"
Oct 30 21:33:09.461554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.461507    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece"} err="failed to get container status \"b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece\": rpc error: code = NotFound desc = could not find container \"b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece\": container with ID starting with b8d4d4876b72120e72fa38b29d46df80e31602c0224ecc2ed53da90af0c57ece not found: ID does not exist"
Oct 30 21:33:09.475711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.475687    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"]
Oct 30 21:33:09.480068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.480049    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-1f1b14/noten-model-1f1b14-kserve-6ff44d869-qvdv8"]
Oct 30 21:33:09.793276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:09.793180    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" path="/var/lib/kubelet/pods/73b2ff52-b6e7-41ca-9bd7-67fb01a95c27/volumes"
Oct 30 21:33:11.662520 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.662486    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"]
Oct 30 21:33:11.663918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.663889    4633 cpu_manager.go:414] "containerMap: removing container" podUID="73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" containerName="main"
Oct 30 21:33:11.663918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.663917    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" containerName="main"
Oct 30 21:33:11.664208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.664192    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b2ff52-b6e7-41ca-9bd7-67fb01a95c27" containerName="main"
Oct 30 21:33:11.673465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.673438    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.688838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.688806    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"]
Oct 30 21:33:11.690138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.690112    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-02ecd7\"/\"explicit-model-02ecd7-kserve-self-signed-certs\""
Oct 30 21:33:11.692782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.690997    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-02ecd7\"/\"kube-root-ca.crt\""
Oct 30 21:33:11.692924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.691083    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-02ecd7\"/\"openshift-service-ca.crt\""
Oct 30 21:33:11.693017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.691134    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-02ecd7\"/\"default-dockercfg-kbwf7\""
Oct 30 21:33:11.728784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.728755    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-model-cache\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.728954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.728915    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.729003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.728962    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-tmp-dir\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.729065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.729008    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xx86\" (UniqueName: \"kubernetes.io/projected/f0231167-bea6-4312-85da-431adc298c62-kube-api-access-9xx86\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.729122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.729097    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-dshm\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.729276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.729260    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-home\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831083 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831043    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831107    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-tmp-dir\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831312    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xx86\" (UniqueName: \"kubernetes.io/projected/f0231167-bea6-4312-85da-431adc298c62-kube-api-access-9xx86\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831345    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-dshm\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831491    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-tmp-dir\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831513    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-home\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831647 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831579    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-model-cache\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831845    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-home\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.831985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.831965    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-model-cache\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.833699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.833654    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-dshm\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.833806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.833765    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:11.842207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:11.842172    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xx86\" (UniqueName: \"kubernetes.io/projected/f0231167-bea6-4312-85da-431adc298c62-kube-api-access-9xx86\") pod \"explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") " pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:12.002146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:12.002039    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:12.325567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:12.325538    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"]
Oct 30 21:33:12.325704 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:33:12.325571    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0231167_bea6_4312_85da_431adc298c62.slice/crio-51d322e5f240f3c1e55a563164aaf46f147527e20e4da1acef030ec9c68a6440 WatchSource:0}: Error finding container 51d322e5f240f3c1e55a563164aaf46f147527e20e4da1acef030ec9c68a6440: Status 404 returned error can't find the container with id 51d322e5f240f3c1e55a563164aaf46f147527e20e4da1acef030ec9c68a6440
Oct 30 21:33:12.500839 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:12.500799    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" event={"ID":"f0231167-bea6-4312-85da-431adc298c62","Type":"ContainerStarted","Data":"68b721d08d9133b0970290d85425b904d6a1428f4c8c117a7ac13e84dc6d7d0e"}
Oct 30 21:33:12.501117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:12.501093    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" event={"ID":"f0231167-bea6-4312-85da-431adc298c62","Type":"ContainerStarted","Data":"51d322e5f240f3c1e55a563164aaf46f147527e20e4da1acef030ec9c68a6440"}
Oct 30 21:33:13.003303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:13.003267    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:22.002572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:22.002532    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:22.095702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:22.095339    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:22.164923 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:22.164323    4633 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-self-signed-certs: secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:22.164923 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:22.164405    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs podName:f0231167-bea6-4312-85da-431adc298c62 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:22.66438413 +0000 UTC m=+1911.014111524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs") pod "explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" (UID: "f0231167-bea6-4312-85da-431adc298c62") : secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:22.230703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:22.214176    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" podStartSLOduration=11.214153277 podStartE2EDuration="11.214153277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:12.5261491 +0000 UTC m=+1900.875876498" watchObservedRunningTime="2026-10-30 21:33:22.214153277 +0000 UTC m=+1910.563880677"
Oct 30 21:33:22.464920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:22.464868    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"]
Oct 30 21:33:22.680691 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:22.679921    4633 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-self-signed-certs: secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:22.680691 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:22.680012    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs podName:f0231167-bea6-4312-85da-431adc298c62 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:23.679993473 +0000 UTC m=+1912.029720868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs") pod "explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" (UID: "f0231167-bea6-4312-85da-431adc298c62") : secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:23.108487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:23.108437    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:23.804436 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:23.804408    4633 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-self-signed-certs: secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:23.804761 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:23.804742    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs podName:f0231167-bea6-4312-85da-431adc298c62 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:25.804718606 +0000 UTC m=+1914.154445993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs") pod "explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" (UID: "f0231167-bea6-4312-85da-431adc298c62") : secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:25.026704 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:25.026649    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:33:25.610894 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:25.610851    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:33:25.611095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:25.610929    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:25.824491 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:25.824460    4633 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-self-signed-certs: secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:25.824818 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:25.824793    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs podName:f0231167-bea6-4312-85da-431adc298c62 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.824771073 +0000 UTC m=+1918.174498459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs") pod "explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" (UID: "f0231167-bea6-4312-85da-431adc298c62") : secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:29.873399 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:29.873360    4633 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-self-signed-certs: secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:29.883035 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:29.874652    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs podName:f0231167-bea6-4312-85da-431adc298c62 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:37.874625314 +0000 UTC m=+1926.224352701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs") pod "explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" (UID: "f0231167-bea6-4312-85da-431adc298c62") : secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:32.446442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.319983    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" podUID="f0231167-bea6-4312-85da-431adc298c62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.216:8000/ready\": context deadline exceeded"
Oct 30 21:33:32.447057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.283000    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:33:32.447057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.280603    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8" podUID="50f5a97c-2b36-48ea-ba42-cee113124243" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:33:32.803983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.803872    4633 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" podUID="4f050773-b095-4cae-b517-d994151796d3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.80:50052: i/o timeout"
Oct 30 21:33:32.804830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.804793    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qcjfz" podUID="eee866b3-a796-49fb-92dc-a4c27f0aa9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:33:32.811149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.811111    4633 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.451664115s: [/var/lib/containers/storage/overlay/98457f07f3aedb4c586c087e4eb10805d3bcfbbefd8c9f34154f15c043b7c8cb/diff /var/log/pods/openshift-ingress_e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd_a07f6b01-10e6-4f92-a40f-87f2d6154725/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 30 21:33:32.812586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.812552    4633 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8" podUID="50f5a97c-2b36-48ea-ba42-cee113124243" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:32.814098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.814073    4633 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.347043911s: [/var/lib/containers/storage/overlay/27c2ea8fe27d85b4362ca3b6a6495b6a84e20e87c8b96149bcb9f8a2338ce6d5/diff /var/log/pods/openshift-ingress_e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7_a9e4e393-82cf-44c5-9247-321be52321a3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 30 21:33:32.825469 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:32.824227    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-lj99j" podUID="e1d828e0-1eea-42a3-b932-223813212485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:33:33.020825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:33.020753    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"]
Oct 30 21:33:33.103379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:33.103295    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"]
Oct 30 21:33:34.108861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:34.108783    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:33:34.108861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:34.108848    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:36.100592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.100561    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"]
Oct 30 21:33:36.124547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.123434    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"]
Oct 30 21:33:36.124547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.123556    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.214829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.214801    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.214829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.214837    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.215086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.214915    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.215086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.214993    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv5v2\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-kube-api-access-vv5v2\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.215086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.215024    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.215265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.215120    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.215265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.215154    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.215265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.215251    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.215478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.215302    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317319    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317441    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317478    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317535    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317582    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317775    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317801    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317861    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.317946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317891    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.318127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.317983    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv5v2\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-kube-api-access-vv5v2\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.318258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.318233    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.318323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.318268    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.318743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.318715    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.318839 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.318752    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.363196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.363068    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.363465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.363444    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv5v2\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-kube-api-access-vv5v2\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.363568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.363464    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.363913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.363890    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:36.456735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:36.456663    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:37.405801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:37.405769    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"]
Oct 30 21:33:37.408124 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:33:37.408097    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cb03b6_5f8c_4e62_9bb8_4285acb19238.slice/crio-4ceb37acfa39ff721baa1fa31d31bdb055d0d2edd7b3ba26e82f99a93814ccd1 WatchSource:0}: Error finding container 4ceb37acfa39ff721baa1fa31d31bdb055d0d2edd7b3ba26e82f99a93814ccd1: Status 404 returned error can't find the container with id 4ceb37acfa39ff721baa1fa31d31bdb055d0d2edd7b3ba26e82f99a93814ccd1
Oct 30 21:33:37.412581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:37.412539    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:37.412732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:37.412633    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:37.412811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:37.412731    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:37.956761 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:37.956725    4633 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-self-signed-certs: secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:37.956971 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:37.956821    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs podName:f0231167-bea6-4312-85da-431adc298c62 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:53.95679589 +0000 UTC m=+1942.306523270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs") pod "explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" (UID: "f0231167-bea6-4312-85da-431adc298c62") : secret "explicit-model-02ecd7-kserve-self-signed-certs" not found
Oct 30 21:33:37.962527 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:37.962490    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" event={"ID":"50cb03b6-5f8c-4e62-9bb8-4285acb19238","Type":"ContainerStarted","Data":"352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a"}
Oct 30 21:33:37.962728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:37.962559    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" event={"ID":"50cb03b6-5f8c-4e62-9bb8-4285acb19238","Type":"ContainerStarted","Data":"4ceb37acfa39ff721baa1fa31d31bdb055d0d2edd7b3ba26e82f99a93814ccd1"}
Oct 30 21:33:37.991952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:37.991892    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podStartSLOduration=1.9918737549999999 podStartE2EDuration="1.991873755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:37.982194504 +0000 UTC m=+1926.331921901" watchObservedRunningTime="2026-10-30 21:33:37.991873755 +0000 UTC m=+1926.341601152"
Oct 30 21:33:38.021878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.021831    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5" podUID="d73bf3bf-0b3d-417a-bf59-b86906557bad" containerName="payload-processing" containerID="cri-o://ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e" gracePeriod=25
Oct 30 21:33:38.104457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.104338    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g" podUID="2a701134-7e5f-4b2b-81f5-6f1d1925a863" containerName="payload-pre-processing" containerID="cri-o://8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb" gracePeriod=25
Oct 30 21:33:38.458655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.457362    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:38.459135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.458755    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 21:33:38.459135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.458829    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 21:33:38.525597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.525197    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:33:38.564778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.563916    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73bf3bf-0b3d-417a-bf59-b86906557bad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf3bf-0b3d-417a-bf59-b86906557bad-plugins-config-volume\") pod \"d73bf3bf-0b3d-417a-bf59-b86906557bad\" (UID: \"d73bf3bf-0b3d-417a-bf59-b86906557bad\") "
Oct 30 21:33:38.564778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.564171    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73bf3bf-0b3d-417a-bf59-b86906557bad-kube-api-access-krx7f\" (UniqueName: \"kubernetes.io/projected/d73bf3bf-0b3d-417a-bf59-b86906557bad-kube-api-access-krx7f\") pod \"d73bf3bf-0b3d-417a-bf59-b86906557bad\" (UID: \"d73bf3bf-0b3d-417a-bf59-b86906557bad\") "
Oct 30 21:33:38.569093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.569050    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73bf3bf-0b3d-417a-bf59-b86906557bad-plugins-config-volume" pod "d73bf3bf-0b3d-417a-bf59-b86906557bad" (UID: "d73bf3bf-0b3d-417a-bf59-b86906557bad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:38.569615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.569586    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73bf3bf-0b3d-417a-bf59-b86906557bad-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:38.609246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.599077    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73bf3bf-0b3d-417a-bf59-b86906557bad-kube-api-access-krx7f" pod "d73bf3bf-0b3d-417a-bf59-b86906557bad" (UID: "d73bf3bf-0b3d-417a-bf59-b86906557bad"). InnerVolumeSpecName "kube-api-access-krx7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:38.679096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.678123    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krx7f\" (UniqueName: \"kubernetes.io/projected/d73bf3bf-0b3d-417a-bf59-b86906557bad-kube-api-access-krx7f\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:38.705195 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.704915    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:33:38.781774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.779148    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a701134-7e5f-4b2b-81f5-6f1d1925a863-kube-api-access-vsj7k\" (UniqueName: \"kubernetes.io/projected/2a701134-7e5f-4b2b-81f5-6f1d1925a863-kube-api-access-vsj7k\") pod \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\" (UID: \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\") "
Oct 30 21:33:38.781774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.779243    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a701134-7e5f-4b2b-81f5-6f1d1925a863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a701134-7e5f-4b2b-81f5-6f1d1925a863-plugins-config-volume\") pod \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\" (UID: \"2a701134-7e5f-4b2b-81f5-6f1d1925a863\") "
Oct 30 21:33:38.781774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.780630    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a701134-7e5f-4b2b-81f5-6f1d1925a863-plugins-config-volume" pod "2a701134-7e5f-4b2b-81f5-6f1d1925a863" (UID: "2a701134-7e5f-4b2b-81f5-6f1d1925a863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:38.785114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.784511    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a701134-7e5f-4b2b-81f5-6f1d1925a863-kube-api-access-vsj7k" pod "2a701134-7e5f-4b2b-81f5-6f1d1925a863" (UID: "2a701134-7e5f-4b2b-81f5-6f1d1925a863"). InnerVolumeSpecName "kube-api-access-vsj7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:38.842940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.832910    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:33:38.842940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.832981    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:38.882717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.882610    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsj7k\" (UniqueName: \"kubernetes.io/projected/2a701134-7e5f-4b2b-81f5-6f1d1925a863-kube-api-access-vsj7k\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:38.882717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.882647    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a701134-7e5f-4b2b-81f5-6f1d1925a863-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:38.973743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.973700    4633 generic.go:356] "Generic (PLEG): container finished" podID="2a701134-7e5f-4b2b-81f5-6f1d1925a863" containerID="8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb" exitCode=0
Oct 30 21:33:38.973931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.973749    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"
Oct 30 21:33:38.973931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.973744    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g" event={"ID":"2a701134-7e5f-4b2b-81f5-6f1d1925a863","Type":"ContainerDied","Data":"8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb"}
Oct 30 21:33:38.973931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.973875    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g" event={"ID":"2a701134-7e5f-4b2b-81f5-6f1d1925a863","Type":"ContainerDied","Data":"689bdcb00ce0f18a27a719fefcb72cf447da6d0aa85e0faab9f22962f9fe0992"}
Oct 30 21:33:38.973931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.973916    4633 scope.go:122] "RemoveContainer" containerID="8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb"
Oct 30 21:33:38.982525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.975774    4633 generic.go:356] "Generic (PLEG): container finished" podID="d73bf3bf-0b3d-417a-bf59-b86906557bad" containerID="ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e" exitCode=0
Oct 30 21:33:38.982525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.975821    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"
Oct 30 21:33:38.982525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.975986    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5" event={"ID":"d73bf3bf-0b3d-417a-bf59-b86906557bad","Type":"ContainerDied","Data":"ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e"}
Oct 30 21:33:38.982525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.976038    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5" event={"ID":"d73bf3bf-0b3d-417a-bf59-b86906557bad","Type":"ContainerDied","Data":"81be8b55f46d1c76e24e26b1c11763496f93fcc420d339e8a63d9b555563e547"}
Oct 30 21:33:38.992453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.992434    4633 scope.go:122] "RemoveContainer" containerID="8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb"
Oct 30 21:33:38.992833 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:38.992813    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb\": container with ID starting with 8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb not found: ID does not exist" containerID="8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb"
Oct 30 21:33:38.992920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.992846    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb"} err="failed to get container status \"8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb\": rpc error: code = NotFound desc = could not find container \"8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb\": container with ID starting with 8713482a606d7c691d27227204df57bf4fbd30ce1326c9610ba4fbc63f7caabb not found: ID does not exist"
Oct 30 21:33:38.992920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:38.992863    4633 scope.go:122] "RemoveContainer" containerID="ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e"
Oct 30 21:33:39.007443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.007419    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"]
Oct 30 21:33:39.044903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.044843    4633 scope.go:122] "RemoveContainer" containerID="ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e"
Oct 30 21:33:39.045088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.044974    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-02ecd7-78db54dcf-d9g4g"]
Oct 30 21:33:39.045388 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:39.045321    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e\": container with ID starting with ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e not found: ID does not exist" containerID="ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e"
Oct 30 21:33:39.045388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.045365    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e"} err="failed to get container status \"ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e\": rpc error: code = NotFound desc = could not find container \"ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e\": container with ID starting with ff418175d5b563f2ccdc910ed93d1041a5e93a1fdd74ca7979aa1858458a349e not found: ID does not exist"
Oct 30 21:33:39.050039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.049974    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"]
Oct 30 21:33:39.055194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.055170    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-02ecd7-574ddc7c67-hr2q5"]
Oct 30 21:33:39.097709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.096430    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"]
Oct 30 21:33:39.098258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.098232    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2a701134-7e5f-4b2b-81f5-6f1d1925a863" containerName="payload-pre-processing"
Oct 30 21:33:39.098422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.098406    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a701134-7e5f-4b2b-81f5-6f1d1925a863" containerName="payload-pre-processing"
Oct 30 21:33:39.098544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.098533    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d73bf3bf-0b3d-417a-bf59-b86906557bad" containerName="payload-processing"
Oct 30 21:33:39.098633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.098621    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73bf3bf-0b3d-417a-bf59-b86906557bad" containerName="payload-processing"
Oct 30 21:33:39.099408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.099385    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a701134-7e5f-4b2b-81f5-6f1d1925a863" containerName="payload-pre-processing"
Oct 30 21:33:39.099555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.099542    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73bf3bf-0b3d-417a-bf59-b86906557bad" containerName="payload-processing"
Oct 30 21:33:39.189031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.188986    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"]
Oct 30 21:33:39.190227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.190189    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.207830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.207734    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a378a2-openshift-default-dockercfg-4ch9v\""
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.350919    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed24f34-9426-459c-8d08-28024297f237-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.351015    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.351084    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.351254    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.351286    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.351379    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.351456    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.351508    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9xk6\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-kube-api-access-m9xk6\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.387847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.351540    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed24f34-9426-459c-8d08-28024297f237-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.458203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.447362    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" podUID="f0231167-bea6-4312-85da-431adc298c62" containerName="main" containerID="cri-o://68b721d08d9133b0970290d85425b904d6a1428f4c8c117a7ac13e84dc6d7d0e" gracePeriod=45
Oct 30 21:33:39.499969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.496331    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 21:33:39.499969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.496403    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.526869    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.526914    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.526998    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.527070    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.527115    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9xk6\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-kube-api-access-m9xk6\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.527147    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed24f34-9426-459c-8d08-28024297f237-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.527209    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed24f34-9426-459c-8d08-28024297f237-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.527276    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.527338    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.561695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.544923    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.579720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.569232    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed24f34-9426-459c-8d08-28024297f237-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.579720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.569831    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.579720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.570073    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.579720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.577788    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.590358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.590295    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.591799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.591717    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed24f34-9426-459c-8d08-28024297f237-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.593280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.593203    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.593653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.593609    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9xk6\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-kube-api-access-m9xk6\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.737856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.737795    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:39.807081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.807045    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a701134-7e5f-4b2b-81f5-6f1d1925a863" path="/var/lib/kubelet/pods/2a701134-7e5f-4b2b-81f5-6f1d1925a863/volumes"
Oct 30 21:33:39.808225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.808071    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73bf3bf-0b3d-417a-bf59-b86906557bad" path="/var/lib/kubelet/pods/d73bf3bf-0b3d-417a-bf59-b86906557bad/volumes"
Oct 30 21:33:39.951476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.949570    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" podUID="f0231167-bea6-4312-85da-431adc298c62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.216:8000/ready\": dial tcp 10.133.0.216:8000: connect: connection refused"
Oct 30 21:33:39.994422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.994385    4633 generic.go:356] "Generic (PLEG): container finished" podID="f0231167-bea6-4312-85da-431adc298c62" containerID="68b721d08d9133b0970290d85425b904d6a1428f4c8c117a7ac13e84dc6d7d0e" exitCode=0
Oct 30 21:33:39.995423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:39.995395    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" event={"ID":"f0231167-bea6-4312-85da-431adc298c62","Type":"ContainerDied","Data":"68b721d08d9133b0970290d85425b904d6a1428f4c8c117a7ac13e84dc6d7d0e"}
Oct 30 21:33:40.151583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.151535    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"]
Oct 30 21:33:40.156516 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:33:40.156468    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed24f34_9426_459c_8d08_28024297f237.slice/crio-0e4bf39fbe83d090f668618499812ff14cde693438d07861e5a49f6614f54b0c WatchSource:0}: Error finding container 0e4bf39fbe83d090f668618499812ff14cde693438d07861e5a49f6614f54b0c: Status 404 returned error can't find the container with id 0e4bf39fbe83d090f668618499812ff14cde693438d07861e5a49f6614f54b0c
Oct 30 21:33:40.165088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.164138    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:40.165088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.164235    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:40.165088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.164280    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:40.203060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.203033    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:40.399650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.399620    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-tmp-dir\") pod \"f0231167-bea6-4312-85da-431adc298c62\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") "
Oct 30 21:33:40.399872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.399744    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs\") pod \"f0231167-bea6-4312-85da-431adc298c62\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") "
Oct 30 21:33:40.399872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.399845    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-home\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-home\") pod \"f0231167-bea6-4312-85da-431adc298c62\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") "
Oct 30 21:33:40.400035 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.400009    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-model-cache\") pod \"f0231167-bea6-4312-85da-431adc298c62\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") "
Oct 30 21:33:40.400155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.400107    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0231167-bea6-4312-85da-431adc298c62-kube-api-access-9xx86\" (UniqueName: \"kubernetes.io/projected/f0231167-bea6-4312-85da-431adc298c62-kube-api-access-9xx86\") pod \"f0231167-bea6-4312-85da-431adc298c62\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") "
Oct 30 21:33:40.400203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.400192    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-dshm\") pod \"f0231167-bea6-4312-85da-431adc298c62\" (UID: \"f0231167-bea6-4312-85da-431adc298c62\") "
Oct 30 21:33:40.400274 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.400253    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-tmp-dir" pod "f0231167-bea6-4312-85da-431adc298c62" (UID: "f0231167-bea6-4312-85da-431adc298c62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:40.400472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.400446    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-home" pod "f0231167-bea6-4312-85da-431adc298c62" (UID: "f0231167-bea6-4312-85da-431adc298c62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:40.400537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.400516    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-model-cache" pod "f0231167-bea6-4312-85da-431adc298c62" (UID: "f0231167-bea6-4312-85da-431adc298c62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:40.401127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.401059    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:40.401127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.401086    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:40.401127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.401099    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:40.403347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.403285    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs" pod "f0231167-bea6-4312-85da-431adc298c62" (UID: "f0231167-bea6-4312-85da-431adc298c62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:40.403347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.403298    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-dshm" pod "f0231167-bea6-4312-85da-431adc298c62" (UID: "f0231167-bea6-4312-85da-431adc298c62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:33:40.403723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.403699    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0231167-bea6-4312-85da-431adc298c62-kube-api-access-9xx86" pod "f0231167-bea6-4312-85da-431adc298c62" (UID: "f0231167-bea6-4312-85da-431adc298c62"). InnerVolumeSpecName "kube-api-access-9xx86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:40.457606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.457562    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 21:33:40.457822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.457636    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 21:33:40.502932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.502864    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0231167-bea6-4312-85da-431adc298c62-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:40.502932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.502895    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xx86\" (UniqueName: \"kubernetes.io/projected/f0231167-bea6-4312-85da-431adc298c62-kube-api-access-9xx86\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:40.502932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.502910    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0231167-bea6-4312-85da-431adc298c62-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:40.623695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.621852    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:33:40.623695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:40.621923    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:41.056499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.056460    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" event={"ID":"0ed24f34-9426-459c-8d08-28024297f237","Type":"ContainerStarted","Data":"f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89"}
Oct 30 21:33:41.056775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.056526    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" event={"ID":"0ed24f34-9426-459c-8d08-28024297f237","Type":"ContainerStarted","Data":"0e4bf39fbe83d090f668618499812ff14cde693438d07861e5a49f6614f54b0c"}
Oct 30 21:33:41.059762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.059731    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9" event={"ID":"f0231167-bea6-4312-85da-431adc298c62","Type":"ContainerDied","Data":"51d322e5f240f3c1e55a563164aaf46f147527e20e4da1acef030ec9c68a6440"}
Oct 30 21:33:41.059897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.059779    4633 scope.go:122] "RemoveContainer" containerID="68b721d08d9133b0970290d85425b904d6a1428f4c8c117a7ac13e84dc6d7d0e"
Oct 30 21:33:41.059978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.059922    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"
Oct 30 21:33:41.088077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.088022    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podStartSLOduration=2.088004252 podStartE2EDuration="2.088004252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:41.077019466 +0000 UTC m=+1929.426746867" watchObservedRunningTime="2026-10-30 21:33:41.088004252 +0000 UTC m=+1929.437731649"
Oct 30 21:33:41.104925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.104897    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"]
Oct 30 21:33:41.109400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.109374    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-02ecd7/explicit-model-02ecd7-kserve-86b9cbb5b5-cn8v9"]
Oct 30 21:33:41.458031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.457985    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 21:33:41.458291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.458049    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 21:33:41.741687 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.741087    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:41.752875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.752841    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:41.803411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:41.803378    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0231167-bea6-4312-85da-431adc298c62" path="/var/lib/kubelet/pods/f0231167-bea6-4312-85da-431adc298c62/volumes"
Oct 30 21:33:42.078482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:42.078379    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:42.082551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:42.082524    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:33:42.460882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:42.457661    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 21:33:42.460882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:42.457741    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 21:33:43.457730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:43.457231    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 21:33:43.457730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:43.457303    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 21:33:44.100059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:44.100028    4633 generic.go:356] "Generic (PLEG): container finished" podID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerID="cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649" exitCode=0
Oct 30 21:33:44.100272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:44.100099    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" event={"ID":"54ef3b75-43f3-44ea-a266-7184d31c4331","Type":"ContainerDied","Data":"cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649"}
Oct 30 21:33:44.100716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:44.100691    4633 scope.go:122] "RemoveContainer" containerID="cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649"
Oct 30 21:33:44.102500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:44.102469    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:44.102606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:44.102540    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:44.102606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:44.102577    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:44.464364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:44.458127    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 21:33:44.464364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:44.458190    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.343099    4633 generic.go:356] "Generic (PLEG): container finished" podID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerID="bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233" exitCode=0
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.343255    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" event={"ID":"a9b242af-fb91-4654-9b6a-975c80d2a871","Type":"ContainerDied","Data":"bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233"}
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.343781    4633 scope.go:122] "RemoveContainer" containerID="bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233"
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.349871    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.349967    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.350018    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.352840    4633 generic.go:356] "Generic (PLEG): container finished" podID="a9e4e393-82cf-44c5-9247-321be52321a3" containerID="272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0" exitCode=0
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.353046    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" event={"ID":"a9e4e393-82cf-44c5-9247-321be52321a3","Type":"ContainerDied","Data":"272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0"}
Oct 30 21:33:45.369696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.353540    4633 scope.go:122] "RemoveContainer" containerID="272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0"
Oct 30 21:33:45.370898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.370760    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" event={"ID":"54ef3b75-43f3-44ea-a266-7184d31c4331","Type":"ContainerStarted","Data":"a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893"}
Oct 30 21:33:45.381064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.381026    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:45.381308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.381286    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:45.381441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.381425    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:45.476224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.476189    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:45.478296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.478170    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:45.479886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.479859    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:33:45.500759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.500729    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:33:45.509863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.509828    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:33:45.510024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.509878    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 21:33:45.965286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:45.965247    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:33:46.407704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.406110    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" event={"ID":"a9b242af-fb91-4654-9b6a-975c80d2a871","Type":"ContainerStarted","Data":"ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e"}
Oct 30 21:33:46.430955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.430917    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" event={"ID":"a9e4e393-82cf-44c5-9247-321be52321a3","Type":"ContainerStarted","Data":"801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601"}
Oct 30 21:33:46.506692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.497287    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:33:46.506692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.497349    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 21:33:46.568217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.568183    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:33:46.568625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.568602    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:46.568834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.568807    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:46.794226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.794128    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"]
Oct 30 21:33:46.798267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.798231    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn" podUID="fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" containerName="maas-api" containerID="cri-o://4f496328c1c71355b23832e4bcf77b0d9180200a1a481e135b9d1a4fe7746654" gracePeriod=30
Oct 30 21:33:46.971319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.963994    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:33:46.993095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.993061    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 21:33:46.993414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:46.993369    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 21:33:47.150280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.130197    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn" podUID="fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.212:8443/health\": dial tcp 10.133.0.212:8443: connect: connection refused"
Oct 30 21:33:47.573696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.572857    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:33:47.573696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.572941    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 21:33:47.612606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.612559    4633 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:47.665239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.653533    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:47.665239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.653599    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:47.710901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.710861    4633 generic.go:356] "Generic (PLEG): container finished" podID="fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" containerID="4f496328c1c71355b23832e4bcf77b0d9180200a1a481e135b9d1a4fe7746654" exitCode=0
Oct 30 21:33:47.738698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.711734    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn" event={"ID":"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a","Type":"ContainerDied","Data":"4f496328c1c71355b23832e4bcf77b0d9180200a1a481e135b9d1a4fe7746654"}
Oct 30 21:33:47.926693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.919434    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 21:33:47.926693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:47.919510    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 21:33:48.308276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.300584    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:33:48.464936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.461125    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:33:48.507936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.507891    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:33:48.508278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.508243    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 21:33:48.571978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.571409    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:48.571978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.571472    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:48.592533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.590946    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-maas-api-tls\") pod \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") "
Oct 30 21:33:48.592533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.591135    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-kube-api-access-z8gv2\" (UniqueName: \"kubernetes.io/projected/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-kube-api-access-z8gv2\") pod \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") "
Oct 30 21:33:48.592533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.591180    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-openshift-service-ca\") pod \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") "
Oct 30 21:33:48.592533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.591289    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-metrics-certs\") pod \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\" (UID: \"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a\") "
Oct 30 21:33:48.594282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.594196    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-openshift-service-ca" pod "fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" (UID: "fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:48.632074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.626712    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-maas-api-tls" pod "fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" (UID: "fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:48.632074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.631796    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-metrics-certs" pod "fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" (UID: "fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:48.694405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.694341    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-kube-api-access-z8gv2" pod "fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" (UID: "fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a"). InnerVolumeSpecName "kube-api-access-z8gv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:48.695153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.695125    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8gv2\" (UniqueName: \"kubernetes.io/projected/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-kube-api-access-z8gv2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:48.695320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.695304    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:48.695429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.695417    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:48.695519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.695508    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:48.792884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.790709    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"
Oct 30 21:33:48.795743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.794029    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn" event={"ID":"fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a","Type":"ContainerDied","Data":"956fc071608242ba7a70004238d136dfb88494323f75bbe4974663cb843b9e62"}
Oct 30 21:33:48.795743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.794096    4633 scope.go:122] "RemoveContainer" containerID="4f496328c1c71355b23832e4bcf77b0d9180200a1a481e135b9d1a4fe7746654"
Oct 30 21:33:48.923696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.923163    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 21:33:48.923696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.923323    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 21:33:48.946183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.940409    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"]
Oct 30 21:33:48.954556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:48.954521    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-02ecd7-778bc76548-2dmzn"]
Oct 30 21:33:49.518708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:49.514924    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:33:49.518708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:49.514991    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 21:33:49.568353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:49.568263    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:49.568605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:49.568578    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:49.932639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:49.932602    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 21:33:49.933585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:49.933318    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 21:33:50.270750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:50.270485    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" path="/var/lib/kubelet/pods/fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a/volumes"
Oct 30 21:33:50.500608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:50.498044    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:33:50.500608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:50.498108    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 21:33:50.593760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:50.577524    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:50.593760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:50.577587    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:50.914045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:50.914006    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 21:33:50.914308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:50.914282    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 21:33:51.507243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:51.505224    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:33:51.507243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:51.505287    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 21:33:51.574178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:51.574143    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:51.574493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:51.574449    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:51.924693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:51.917063    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 21:33:51.924693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:51.917125    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 21:33:52.499694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:52.499104    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 21:33:52.499694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:52.499167    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 21:33:52.570637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:52.570601    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:52.571292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:52.571241    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:52.778584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:52.769808    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:33:52.932077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:52.932040    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 21:33:52.932361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:52.932331    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 21:33:53.571051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:53.571008    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:53.574743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:53.574702    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:53.917582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:53.917547    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused" start-of-body=
Oct 30 21:33:53.917904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:53.917869    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.207:15021/healthz/ready\": dial tcp 10.133.0.207:15021: connect: connection refused"
Oct 30 21:33:54.506512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.506208    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:33:54.506512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.506263    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:54.568433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.568393    4633 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:33:54.568639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.568453    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused"
Oct 30 21:33:54.570128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.570099    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:33:54.604251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.604212    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:33:54.626317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.625577    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"]
Oct 30 21:33:54.628481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.628451    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f0231167-bea6-4312-85da-431adc298c62" containerName="main"
Oct 30 21:33:54.628687 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.628653    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0231167-bea6-4312-85da-431adc298c62" containerName="main"
Oct 30 21:33:54.628850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.628838    4633 cpu_manager.go:414] "containerMap: removing container" podUID="fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" containerName="maas-api"
Oct 30 21:33:54.628935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.628924    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" containerName="maas-api"
Oct 30 21:33:54.629547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.629522    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0231167-bea6-4312-85da-431adc298c62" containerName="main"
Oct 30 21:33:54.629731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.629716    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd4f737-8cfd-4f31-b2ae-f5c40ff66f8a" containerName="maas-api"
Oct 30 21:33:54.655652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.655621    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.675930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.665566    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-eceb81ede848\""
Oct 30 21:33:54.698454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.698418    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bb88\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-kube-api-access-5bb88\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.712693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.707081    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.712693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.707302    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.712693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.707412    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.712693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.708546    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"]
Oct 30 21:33:54.809254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.809147    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.809645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.809621    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.809877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.809858    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.810195 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.810174    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bb88\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-kube-api-access-5bb88\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.810440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.810183    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.810603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.810574    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.814794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.814766    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:54.819438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:54.819411    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bb88\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-kube-api-access-5bb88\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:55.033439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:55.033163    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:55.079117 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:33:55.078276    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:33:55.199646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:55.199597    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"]
Oct 30 21:33:55.199990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:55.199962    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy" containerID="cri-o://ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e" gracePeriod=30
Oct 30 21:33:55.628060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:55.571162    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"]
Oct 30 21:33:55.685543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:55.685506    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:33:55.840798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:55.840768    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:33:55.842891 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:55.842867    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:33:56.691335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:56.691289    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d" event={"ID":"3f7a4463-2ca0-4834-a81e-c9eb26bf4392","Type":"ContainerStarted","Data":"b3739d4df6ea7065974c26a2bfbe9eff78c462aa91328a030f08431363222ca2"}
Oct 30 21:33:56.691797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:56.691704    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:33:56.691797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:56.691750    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d" event={"ID":"3f7a4463-2ca0-4834-a81e-c9eb26bf4392","Type":"ContainerStarted","Data":"b1521a239122b54decc3242e4e062843de38e7e88508a18ded04e2ae32d15c9f"}
Oct 30 21:33:56.715594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:56.715544    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d" podStartSLOduration=2.715529874 podStartE2EDuration="2.715529874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:56.71293571 +0000 UTC m=+1945.062663107" watchObservedRunningTime="2026-10-30 21:33:56.715529874 +0000 UTC m=+1945.065257303"
Oct 30 21:33:57.613162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.613124    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"]
Oct 30 21:33:57.619554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.619528    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.628782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.628758    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8072a0-openshift-default-dockercfg-2l5wp\""
Oct 30 21:33:57.641082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.641059    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"]
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.876866    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-token\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.877021    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-certs\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.877092    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5fxv\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-kube-api-access-j5fxv\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.877128    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istiod-ca-cert\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.877172    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-data\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.877222    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-podinfo\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.877495    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-socket\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.877542    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-credential-socket\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.877749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.877604    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-envoy\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.996696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996407    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-socket\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.996696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996472    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-credential-socket\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.996696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996527    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-envoy\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.996696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996569    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-token\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.996696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996710    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-certs\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.997308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996754    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5fxv\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-kube-api-access-j5fxv\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.997308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996786    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istiod-ca-cert\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.997308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996825    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-data\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:57.997308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:57.996873    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-podinfo\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.007902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.003289    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-socket\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.007902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.003831    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-credential-socket\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.007902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.004462    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istiod-ca-cert\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.007902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.004769    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-data\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.027172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.009087    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-podinfo\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.030067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.029964    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-certs\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.034611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.033120    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-envoy\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.034611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.034526    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-token\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.046916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.046840    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5fxv\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-kube-api-access-j5fxv\") pod \"e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") " pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.239822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.239783    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:58.591813 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:33:58.591780    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22001a2_054d_40d7_b536_4f1c26d5dbe2.slice/crio-07b379c9217ff547819d71709d34fc0d5d1d47e9a3e85e1924b6f5c67208b545 WatchSource:0}: Error finding container 07b379c9217ff547819d71709d34fc0d5d1d47e9a3e85e1924b6f5c67208b545: Status 404 returned error can't find the container with id 07b379c9217ff547819d71709d34fc0d5d1d47e9a3e85e1924b6f5c67208b545
Oct 30 21:33:58.592164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.592145    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"]
Oct 30 21:33:58.594781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.594749    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:58.594877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.594815    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:58.594877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.594843    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:33:58.712821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.712743    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" event={"ID":"b22001a2-054d-40d7-b536-4f1c26d5dbe2","Type":"ContainerStarted","Data":"eb0f5a34c8e19241f4940518fa62ea118c39e0556508cba1651e7e7033ab937e"}
Oct 30 21:33:58.712821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.712823    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" event={"ID":"b22001a2-054d-40d7-b536-4f1c26d5dbe2","Type":"ContainerStarted","Data":"07b379c9217ff547819d71709d34fc0d5d1d47e9a3e85e1924b6f5c67208b545"}
Oct 30 21:33:58.739020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:58.738961    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podStartSLOduration=1.73894726 podStartE2EDuration="1.73894726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:58.733359548 +0000 UTC m=+1947.083086944" watchObservedRunningTime="2026-10-30 21:33:58.73894726 +0000 UTC m=+1947.088674717"
Oct 30 21:33:59.240077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:59.240039    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:59.246221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:59.246198    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:59.748234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:59.747097    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:33:59.757915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:33:59.757882    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:34:01.026623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.026561    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:34:01.080853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.080810    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-token\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-token\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.081061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.080933    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-podinfo\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.081061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.081026    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-data\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.081181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.081094    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-kube-api-access-xmkxg\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-kube-api-access-xmkxg\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.081234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.081181    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-socket\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.081234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.081227    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-envoy\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.081340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.081283    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-certs\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.081396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.081363    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-credential-socket\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.081480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.081462    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b242af-fb91-4654-9b6a-975c80d2a871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b242af-fb91-4654-9b6a-975c80d2a871-istiod-ca-cert\") pod \"a9b242af-fb91-4654-9b6a-975c80d2a871\" (UID: \"a9b242af-fb91-4654-9b6a-975c80d2a871\") "
Oct 30 21:34:01.082595 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.082564    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b242af-fb91-4654-9b6a-975c80d2a871-istiod-ca-cert" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:01.082872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.082848    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-data" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:01.083716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.083689    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-token" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:01.083970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.083941    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-credential-socket" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:01.083970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.083963    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-socket" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:01.084380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.084355    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-certs" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:01.084623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.084597    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-podinfo" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:01.086066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.086028    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-kube-api-access-xmkxg" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "kube-api-access-xmkxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:01.087772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.087749    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-envoy" pod "a9b242af-fb91-4654-9b6a-975c80d2a871" (UID: "a9b242af-fb91-4654-9b6a-975c80d2a871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:01.184227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184196    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b242af-fb91-4654-9b6a-975c80d2a871-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.184227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184226    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.184227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184237    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.184510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184247    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.184510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184256    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmkxg\" (UniqueName: \"kubernetes.io/projected/a9b242af-fb91-4654-9b6a-975c80d2a871-kube-api-access-xmkxg\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.184510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184267    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.184510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184276    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.184510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184284    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.184510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.184292    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b242af-fb91-4654-9b6a-975c80d2a871-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.767429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.767396    4633 generic.go:356] "Generic (PLEG): container finished" podID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerID="ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e" exitCode=0
Oct 30 21:34:01.767646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.767446    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"
Oct 30 21:34:01.767646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.767477    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" event={"ID":"a9b242af-fb91-4654-9b6a-975c80d2a871","Type":"ContainerDied","Data":"ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e"}
Oct 30 21:34:01.767646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.767533    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf" event={"ID":"a9b242af-fb91-4654-9b6a-975c80d2a871","Type":"ContainerDied","Data":"acd1a941a949593c56d441befa1a655b83ac5c67d0b04724fa44498123c064e1"}
Oct 30 21:34:01.767646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.767556    4633 scope.go:122] "RemoveContainer" containerID="ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e"
Oct 30 21:34:01.780848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.780827    4633 scope.go:122] "RemoveContainer" containerID="bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233"
Oct 30 21:34:01.804905 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.804872    4633 scope.go:122] "RemoveContainer" containerID="ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e"
Oct 30 21:34:01.808150 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:34:01.808110    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e\": container with ID starting with ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e not found: ID does not exist" containerID="ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e"
Oct 30 21:34:01.808300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.808161    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e"} err="failed to get container status \"ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e\": rpc error: code = NotFound desc = could not find container \"ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e\": container with ID starting with ec24b64b1670e4c9297401e7dafe0ee2da7da5bc777fb36e353caccc7d9bc79e not found: ID does not exist"
Oct 30 21:34:01.808300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.808192    4633 scope.go:122] "RemoveContainer" containerID="bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233"
Oct 30 21:34:01.808604 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:34:01.808569    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233\": container with ID starting with bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233 not found: ID does not exist" containerID="bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233"
Oct 30 21:34:01.808697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.808617    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233"} err="failed to get container status \"bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233\": rpc error: code = NotFound desc = could not find container \"bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233\": container with ID starting with bd6ee8849addecb3466a2e58a60bca3b38c61ea705726b06ac541e3dfd3c7233 not found: ID does not exist"
Oct 30 21:34:01.808697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.808642    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"]
Oct 30 21:34:01.809441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:01.809417    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-xhblf"]
Oct 30 21:34:03.793052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:03.793022    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" path="/var/lib/kubelet/pods/a9b242af-fb91-4654-9b6a-975c80d2a871/volumes"
Oct 30 21:34:07.704153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:07.704125    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:34:07.773276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:07.773243    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"]
Oct 30 21:34:12.774029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:12.773977    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859" podUID="c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" containerName="payload-processing" containerID="cri-o://9ad1b1262bf20b9544d4208ba0f641328167c3e69c931cd3e348031a8a3abee1" gracePeriod=25
Oct 30 21:34:12.913018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:12.912977    4633 generic.go:356] "Generic (PLEG): container finished" podID="c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" containerID="9ad1b1262bf20b9544d4208ba0f641328167c3e69c931cd3e348031a8a3abee1" exitCode=0
Oct 30 21:34:12.913154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:12.913040    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859" event={"ID":"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa","Type":"ContainerDied","Data":"9ad1b1262bf20b9544d4208ba0f641328167c3e69c931cd3e348031a8a3abee1"}
Oct 30 21:34:13.082183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.082159    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:34:13.092847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.092823    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-routing-overlay-volume\") pod \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") "
Oct 30 21:34:13.092986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.092874    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-plugins-config-volume\") pod \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") "
Oct 30 21:34:13.093036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.093004    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-provider-credentials\") pod \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") "
Oct 30 21:34:13.093394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.093081    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-kube-api-access-mgvnl\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-kube-api-access-mgvnl\") pod \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\" (UID: \"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa\") "
Oct 30 21:34:13.093394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.093161    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-routing-overlay-volume" pod "c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" (UID: "c25c4119-4cef-4dc3-a72a-d0b38d7b38fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:13.093394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.093232    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-plugins-config-volume" pod "c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" (UID: "c25c4119-4cef-4dc3-a72a-d0b38d7b38fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:13.093812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.093791    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:13.093880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.093819    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:13.095482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.095455    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-provider-credentials" pod "c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" (UID: "c25c4119-4cef-4dc3-a72a-d0b38d7b38fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:13.095589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.095526    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-kube-api-access-mgvnl" pod "c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" (UID: "c25c4119-4cef-4dc3-a72a-d0b38d7b38fa"). InnerVolumeSpecName "kube-api-access-mgvnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:13.195586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.195556    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:13.195586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.195584    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvnl\" (UniqueName: \"kubernetes.io/projected/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa-kube-api-access-mgvnl\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:13.922703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.922639    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"
Oct 30 21:34:13.923189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.922646    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859" event={"ID":"c25c4119-4cef-4dc3-a72a-d0b38d7b38fa","Type":"ContainerDied","Data":"168cf1521258e7ac4d50e6c3db32214e2d77903fa19d7196444436514465a1c0"}
Oct 30 21:34:13.923189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.922832    4633 scope.go:122] "RemoveContainer" containerID="9ad1b1262bf20b9544d4208ba0f641328167c3e69c931cd3e348031a8a3abee1"
Oct 30 21:34:13.942758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.942733    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"]
Oct 30 21:34:13.948087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:13.948054    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754tq859"]
Oct 30 21:34:15.794629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:15.794599    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" path="/var/lib/kubelet/pods/c25c4119-4cef-4dc3-a72a-d0b38d7b38fa/volumes"
Oct 30 21:34:18.369332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.369295    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"]
Oct 30 21:34:18.371551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.371521    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy"
Oct 30 21:34:18.372223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.371553    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy"
Oct 30 21:34:18.372223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.371609    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" containerName="payload-processing"
Oct 30 21:34:18.372223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.371619    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" containerName="payload-processing"
Oct 30 21:34:18.372223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.371691    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy"
Oct 30 21:34:18.372223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.371700    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy"
Oct 30 21:34:18.372223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.371976    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy"
Oct 30 21:34:18.372223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.371995    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c25c4119-4cef-4dc3-a72a-d0b38d7b38fa" containerName="payload-processing"
Oct 30 21:34:18.391499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.391472    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.393383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.393355    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"]
Oct 30 21:34:18.399186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.399164    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8072a0-serving-cert\""
Oct 30 21:34:18.502336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.502297    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-metrics-certs\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.502507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.502451    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.502507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.502476    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/656dbefb-91f4-48ce-afe3-742b52686788-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.502661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.502646    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9lw\" (UniqueName: \"kubernetes.io/projected/656dbefb-91f4-48ce-afe3-742b52686788-kube-api-access-pf9lw\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.605361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.605309    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9lw\" (UniqueName: \"kubernetes.io/projected/656dbefb-91f4-48ce-afe3-742b52686788-kube-api-access-pf9lw\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.605791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.605766    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-metrics-certs\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.606040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.606021    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.606176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.606161    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/656dbefb-91f4-48ce-afe3-742b52686788-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.607245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.607224    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/656dbefb-91f4-48ce-afe3-742b52686788-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.610197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.610173    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.612074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.612048    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-metrics-certs\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.616422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.616392    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9lw\" (UniqueName: \"kubernetes.io/projected/656dbefb-91f4-48ce-afe3-742b52686788-kube-api-access-pf9lw\") pod \"maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.708245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.708215    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:18.794288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.793346    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"]
Oct 30 21:34:18.794288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:18.794033    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy" containerID="cri-o://6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f" gracePeriod=30
Oct 30 21:34:19.183404 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:19.183356    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"]
Oct 30 21:34:20.108992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:20.108952    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj" event={"ID":"656dbefb-91f4-48ce-afe3-742b52686788","Type":"ContainerStarted","Data":"81073fc4516b1d92a45f9cf767e99323f46df047cf42f55001df44d8787ce657"}
Oct 30 21:34:20.109464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:20.109024    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:20.109464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:20.109051    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj" event={"ID":"656dbefb-91f4-48ce-afe3-742b52686788","Type":"ContainerStarted","Data":"10408b2800b19c97a46e5fdb2ae783e76e49e67dca786842d7b2fd38c5bc4565"}
Oct 30 21:34:20.128864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:20.128808    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj" podStartSLOduration=1.684735219 podStartE2EDuration="2.12879316s" totalImagesPullingTime="444.057941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:20.125963292 +0000 UTC m=+1968.475690713" watchObservedRunningTime="2026-10-30 21:34:20.12879316 +0000 UTC m=+1968.478520556"
Oct 30 21:34:24.105222 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.105194    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:34:24.148051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148004    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-data\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148152    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-kube-api-access-dhntf\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-kube-api-access-dhntf\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148187    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-socket\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148242    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-podinfo\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148267    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-token\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-token\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148281    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-data" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:24.148442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148355    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-envoy\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148391    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istiod-ca-cert\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148486    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-credential-socket\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148515    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-socket" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:24.148716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148602    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-certs\") pod \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\" (UID: \"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887\") "
Oct 30 21:34:24.148876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148848    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-credential-socket" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:24.148956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.148874    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istiod-ca-cert" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:24.149294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.149269    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-certs" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:24.149566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.149402    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.149566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.149418    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.149566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.149428    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.149566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.149437    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.151005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.150982    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-kube-api-access-dhntf" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "kube-api-access-dhntf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:24.151148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.151128    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-token" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:24.151148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.151133    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-podinfo" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:24.151231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.151217    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-envoy" pod "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" (UID: "4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:24.204598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.204559    4633 generic.go:356] "Generic (PLEG): container finished" podID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerID="6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f" exitCode=0
Oct 30 21:34:24.204826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.204609    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"
Oct 30 21:34:24.204826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.204641    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" event={"ID":"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887","Type":"ContainerDied","Data":"6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f"}
Oct 30 21:34:24.204826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.204760    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d" event={"ID":"4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887","Type":"ContainerDied","Data":"665d6aac6a84f16a84f7b99d7db35f98a361f4843a92cad43b3c6b6d2db88d84"}
Oct 30 21:34:24.204826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.204786    4633 scope.go:122] "RemoveContainer" containerID="6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f"
Oct 30 21:34:24.217933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.217895    4633 scope.go:122] "RemoveContainer" containerID="6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f"
Oct 30 21:34:24.218269 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:34:24.218241    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f\": container with ID starting with 6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f not found: ID does not exist" containerID="6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f"
Oct 30 21:34:24.218346 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.218282    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f"} err="failed to get container status \"6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f\": rpc error: code = NotFound desc = could not find container \"6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f\": container with ID starting with 6bbc15779fa41ba831737e1865de82077ab57d3b3fd85ddde71c0c78183b4f0f not found: ID does not exist"
Oct 30 21:34:24.251022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.250977    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.251022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.251025    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhntf\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-kube-api-access-dhntf\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.251194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.251043    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.251194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.251057    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.251194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.251071    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:24.255080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.255053    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"]
Oct 30 21:34:24.275992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:24.275958    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-ksd8d"]
Oct 30 21:34:25.015007 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:34:25.014960    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:34:25.793019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:25.792983    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" path="/var/lib/kubelet/pods/4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887/volumes"
Oct 30 21:34:26.038776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.038718    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"]
Oct 30 21:34:26.042505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.039972    4633 cpu_manager.go:414] "containerMap: removing container" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy"
Oct 30 21:34:26.042505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.039995    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy"
Oct 30 21:34:26.042505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.040217    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c66e81d-a2b5-4c15-9bfc-9e65d3b0a887" containerName="istio-proxy"
Oct 30 21:34:26.042505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.040235    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b242af-fb91-4654-9b6a-975c80d2a871" containerName="istio-proxy"
Oct 30 21:34:26.046264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.046200    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.046264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.046219    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"]
Oct 30 21:34:26.050074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.050051    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f9e5a7-openshift-default-dockercfg-7hvlf\""
Oct 30 21:34:26.143699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.137582    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:34:26.318749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.285049    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.379709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.363168    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-podinfo\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.379709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.363344    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-token\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.379709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.369246    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9nq5\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-kube-api-access-p9nq5\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.379709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.371603    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-certs\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.379709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.371723    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-data\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.379709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.372724    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-credential-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.393176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.383242    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-envoy\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.393176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.383448    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istiod-ca-cert\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.489190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489136    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-certs\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.489458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489196    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-data\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.489458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489356    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-credential-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.489458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489412    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-envoy\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.489458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489440    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istiod-ca-cert\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489656    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-certs\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489795    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489907    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-podinfo\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489944    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-token\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.489990    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-credential-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.490058    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9nq5\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-kube-api-access-p9nq5\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.490925    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-data\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.491622    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.493382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.492218    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istiod-ca-cert\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.500431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.500257    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-envoy\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.500431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.500349    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-token\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.500907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.500885    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9nq5\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-kube-api-access-p9nq5\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.501526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.501395    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-podinfo\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.665228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.665189    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:26.957283 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:34:26.957253    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3153c1af_b6f5_482d_bcc6_c577c5ad697d.slice/crio-015d5c55010d2fa49afe9304c1c972dee186836eecf142dc0cee68420b9a6b21 WatchSource:0}: Error finding container 015d5c55010d2fa49afe9304c1c972dee186836eecf142dc0cee68420b9a6b21: Status 404 returned error can't find the container with id 015d5c55010d2fa49afe9304c1c972dee186836eecf142dc0cee68420b9a6b21
Oct 30 21:34:26.959547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.959523    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"]
Oct 30 21:34:26.960246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.960215    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:26.960293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.960279    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:26.960326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:26.960310    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:27.408190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:27.408149    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" event={"ID":"3153c1af-b6f5-482d-bcc6-c577c5ad697d","Type":"ContainerStarted","Data":"bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b"}
Oct 30 21:34:27.408373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:27.408237    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" event={"ID":"3153c1af-b6f5-482d-bcc6-c577c5ad697d","Type":"ContainerStarted","Data":"015d5c55010d2fa49afe9304c1c972dee186836eecf142dc0cee68420b9a6b21"}
Oct 30 21:34:27.431761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:27.431648    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podStartSLOduration=2.431634669 podStartE2EDuration="2.431634669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:27.430570342 +0000 UTC m=+1975.780297760" watchObservedRunningTime="2026-10-30 21:34:27.431634669 +0000 UTC m=+1975.781362064"
Oct 30 21:34:27.666070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:27.665985    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:28.672688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:28.672647    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:29.424608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:29.424573    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:29.425655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:29.425636    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:34:31.144618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.144586    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"]
Oct 30 21:34:31.149505 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.149485    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:31.155218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.155187    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f9e5a7\""
Oct 30 21:34:31.155388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.155300    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f9e5a7-dockercfg-8d6wc\""
Oct 30 21:34:31.161557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.161532    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"]
Oct 30 21:34:31.277692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.253569    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppzcm\" (UniqueName: \"kubernetes.io/projected/95f4f687-e025-46fd-85a5-9a93cfef9cc6-kube-api-access-ppzcm\") pod \"payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7\" (UID: \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:31.282693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.278180    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f4f687-e025-46fd-85a5-9a93cfef9cc6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7\" (UID: \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:31.444389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.444322    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f4f687-e025-46fd-85a5-9a93cfef9cc6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7\" (UID: \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:31.446022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.445988    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppzcm\" (UniqueName: \"kubernetes.io/projected/95f4f687-e025-46fd-85a5-9a93cfef9cc6-kube-api-access-ppzcm\") pod \"payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7\" (UID: \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:31.451875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.451846    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f4f687-e025-46fd-85a5-9a93cfef9cc6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7\" (UID: \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:31.488294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.488230    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppzcm\" (UniqueName: \"kubernetes.io/projected/95f4f687-e025-46fd-85a5-9a93cfef9cc6-kube-api-access-ppzcm\") pod \"payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7\" (UID: \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:31.768201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:31.768067    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:32.149772 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:34:32.149740    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f4f687_e025_46fd_85a5_9a93cfef9cc6.slice/crio-a69b8c31933561a9b9096304d4ed1f92ee4078515315e06901d751e41bb878ab WatchSource:0}: Error finding container a69b8c31933561a9b9096304d4ed1f92ee4078515315e06901d751e41bb878ab: Status 404 returned error can't find the container with id a69b8c31933561a9b9096304d4ed1f92ee4078515315e06901d751e41bb878ab
Oct 30 21:34:32.151344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:32.151312    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"]
Oct 30 21:34:32.496133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:32.496095    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7" event={"ID":"95f4f687-e025-46fd-85a5-9a93cfef9cc6","Type":"ContainerStarted","Data":"d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43"}
Oct 30 21:34:32.496352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:32.496162    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7" event={"ID":"95f4f687-e025-46fd-85a5-9a93cfef9cc6","Type":"ContainerStarted","Data":"a69b8c31933561a9b9096304d4ed1f92ee4078515315e06901d751e41bb878ab"}
Oct 30 21:34:32.496352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:32.496203    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:32.545698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:32.545380    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7" podStartSLOduration=1.545359766 podStartE2EDuration="1.545359766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:32.522608141 +0000 UTC m=+1980.872335537" watchObservedRunningTime="2026-10-30 21:34:32.545359766 +0000 UTC m=+1980.895087164"
Oct 30 21:34:40.593019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:40.592989    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:34:43.508237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:43.508208    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:34:47.191696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.187271    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"]
Oct 30 21:34:47.252697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.234551    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.260740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.257259    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"]
Oct 30 21:34:47.283847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.283810    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f1ed97-openshift-default-dockercfg-h5n84\""
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367030    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-istio-token\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367105    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-credential-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367136    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-envoy\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367171    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-data\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367300    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de810876-cea8-4249-b707-7d20744b27ad-istio-podinfo\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367375    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367569    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-certs\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367599    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de810876-cea8-4249-b707-7d20744b27ad-istiod-ca-cert\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.368540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.367639    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwpcr\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-kube-api-access-kwpcr\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.469720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.469620    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-certs\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.469720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.469661    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de810876-cea8-4249-b707-7d20744b27ad-istiod-ca-cert\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.469720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.469708    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwpcr\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-kube-api-access-kwpcr\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.469965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.469853    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-istio-token\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.469965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.469871    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-credential-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.469965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.469887    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-envoy\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.469965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.469905    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-data\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.470094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.469968    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de810876-cea8-4249-b707-7d20744b27ad-istio-podinfo\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.470094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.470009    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.470393 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.470372    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.470856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.470559    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-credential-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.470856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.470809    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-data\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.470856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.470848    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de810876-cea8-4249-b707-7d20744b27ad-istiod-ca-cert\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.471184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.471164    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-certs\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.473265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.473242    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de810876-cea8-4249-b707-7d20744b27ad-istio-podinfo\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.473442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.473421    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-envoy\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.479215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.479197    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-istio-token\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.479778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.479758    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwpcr\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-kube-api-access-kwpcr\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:47.597929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:47.597889    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:48.003427 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:34:48.002927    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde810876_cea8_4249_b707_7d20744b27ad.slice/crio-34ff15e92621493bfa27e5efefeef2f5753e7aa522a89291849b16b37993349d WatchSource:0}: Error finding container 34ff15e92621493bfa27e5efefeef2f5753e7aa522a89291849b16b37993349d: Status 404 returned error can't find the container with id 34ff15e92621493bfa27e5efefeef2f5753e7aa522a89291849b16b37993349d
Oct 30 21:34:48.003427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:48.003116    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"]
Oct 30 21:34:48.006659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:48.006618    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:48.006774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:48.006732    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:48.006816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:48.006782    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:48.084573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:48.084504    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" event={"ID":"de810876-cea8-4249-b707-7d20744b27ad","Type":"ContainerStarted","Data":"34ff15e92621493bfa27e5efefeef2f5753e7aa522a89291849b16b37993349d"}
Oct 30 21:34:49.100346 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:49.100303    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" event={"ID":"de810876-cea8-4249-b707-7d20744b27ad","Type":"ContainerStarted","Data":"a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca"}
Oct 30 21:34:49.126178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:49.126116    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podStartSLOduration=2.126097078 podStartE2EDuration="2.126097078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:49.123201216 +0000 UTC m=+1997.472928612" watchObservedRunningTime="2026-10-30 21:34:49.126097078 +0000 UTC m=+1997.475824477"
Oct 30 21:34:49.598606 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:49.598547    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:49.601026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:49.600723    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 21:34:49.601026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:49.600774    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 21:34:50.598956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:50.598918    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 21:34:50.599380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:50.598983    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 21:34:51.602161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:51.602082    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 21:34:51.602161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:51.602152    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 21:34:52.599053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:52.599013    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 21:34:52.599229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:52.599083    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 21:34:53.599500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:53.599464    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 21:34:53.599938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:53.599516    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 21:34:54.335694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.335140    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"]
Oct 30 21:34:54.479841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.467334    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.491268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.491239    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"]
Oct 30 21:34:54.582151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582118    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.582350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582167    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.582350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582256    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.582435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582367    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.582543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582526    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.582580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582559    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.582634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582584    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qffw8\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-kube-api-access-qffw8\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.582769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582750    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40741b-59ed-4b6d-95df-8b515a02eb61-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.582916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.582900    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.598638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.598560    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 21:34:54.598638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.598620    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 21:34:54.684219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684187    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.684765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684302    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.684765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684323    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.684765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684340    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.684765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684371    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.684765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684477    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.684765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684513    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.684765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684552    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qffw8\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-kube-api-access-qffw8\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.684765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684588    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40741b-59ed-4b6d-95df-8b515a02eb61-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.685252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684978    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.685252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.684994    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.685252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.685019    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.685431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.685268    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.685431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.685311    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40741b-59ed-4b6d-95df-8b515a02eb61-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.687124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.687098    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.687391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.687374    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.698757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.698729    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qffw8\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-kube-api-access-qffw8\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.699121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.699103    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:54.885276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:54.885201    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:55.056690 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:34:55.056635    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:34:55.263643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:55.263614    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"]
Oct 30 21:34:55.264918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:55.264882    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:55.265032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:55.264960    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:55.265032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:55.265005    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:55.502761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:55.502709    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" event={"ID":"df40741b-59ed-4b6d-95df-8b515a02eb61","Type":"ContainerStarted","Data":"1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b"}
Oct 30 21:34:55.502955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:55.502784    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" event={"ID":"df40741b-59ed-4b6d-95df-8b515a02eb61","Type":"ContainerStarted","Data":"8724e3fa659baf1dcecc9b03ccde42f5c24852aa67d03eede388d673ee8babfd"}
Oct 30 21:34:55.599321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:55.599230    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 30 21:34:55.599321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:55.599292    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 30 21:34:56.572761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:56.572700    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" podStartSLOduration=2.57268566 podStartE2EDuration="2.57268566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:56.568001245 +0000 UTC m=+2004.917728641" watchObservedRunningTime="2026-10-30 21:34:56.57268566 +0000 UTC m=+2004.922413053"
Oct 30 21:34:56.605602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:56.605573    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:56.606246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:56.606216    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:56.606773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:56.606754    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:34:56.886199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:56.886122    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:34:56.887516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:56.887492    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:34:56.887655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:56.887549    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 21:34:56.992643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:56.992226    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"]
Oct 30 21:34:57.436708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.427822    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": read tcp 10.133.0.2:57982->10.133.0.218:15021: read: connection reset by peer" start-of-body=
Oct 30 21:34:57.436708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.427895    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": read tcp 10.133.0.2:57982->10.133.0.218:15021: read: connection reset by peer"
Oct 30 21:34:57.548039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.548005    4633 generic.go:356] "Generic (PLEG): container finished" podID="0ed24f34-9426-459c-8d08-28024297f237" containerID="f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89" exitCode=0
Oct 30 21:34:57.548231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.548082    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" event={"ID":"0ed24f34-9426-459c-8d08-28024297f237","Type":"ContainerDied","Data":"f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89"}
Oct 30 21:34:57.549121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.549100    4633 scope.go:122] "RemoveContainer" containerID="f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89"
Oct 30 21:34:57.551617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.551582    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:57.551727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.551698    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:57.551788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.551768    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:34:57.885819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.885788    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:34:57.886278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:57.885850    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 21:34:58.573317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:58.573280    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" event={"ID":"0ed24f34-9426-459c-8d08-28024297f237","Type":"ContainerStarted","Data":"576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b"}
Oct 30 21:34:58.749694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:58.742879    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:34:58.749694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:58.743387    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 21:34:58.749694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:58.743450    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 21:34:58.891761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:58.891076    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:34:58.891761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:58.891137    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 21:34:59.740655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:59.740616    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 21:34:59.745776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:59.745739    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 21:34:59.885895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:59.885859    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:34:59.886186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:34:59.886157    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 21:35:00.516703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.510112    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": read tcp 10.133.0.2:55484->10.133.0.220:15021: read: connection reset by peer" start-of-body=
Oct 30 21:35:00.516703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.510177    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": read tcp 10.133.0.2:55484->10.133.0.220:15021: read: connection reset by peer"
Oct 30 21:35:00.669786 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.669696    4633 generic.go:356] "Generic (PLEG): container finished" podID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerID="eb0f5a34c8e19241f4940518fa62ea118c39e0556508cba1651e7e7033ab937e" exitCode=0
Oct 30 21:35:00.669978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.669814    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" event={"ID":"b22001a2-054d-40d7-b536-4f1c26d5dbe2","Type":"ContainerDied","Data":"eb0f5a34c8e19241f4940518fa62ea118c39e0556508cba1651e7e7033ab937e"}
Oct 30 21:35:00.670354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.670339    4633 scope.go:122] "RemoveContainer" containerID="eb0f5a34c8e19241f4940518fa62ea118c39e0556508cba1651e7e7033ab937e"
Oct 30 21:35:00.672444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.672408    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:35:00.672532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.672504    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:35:00.672579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.672538    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:35:00.772133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.772041    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 21:35:00.772133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.772109    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 21:35:00.886178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.885952    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:35:00.886178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:00.886061    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 21:35:01.680503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:01.680464    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" event={"ID":"b22001a2-054d-40d7-b536-4f1c26d5dbe2","Type":"ContainerStarted","Data":"7b505ebd678d649e3efb9fde0339d6333775ae755157d6e03080fc9249a9ff84"}
Oct 30 21:35:01.779051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:01.779018    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 21:35:01.779351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:01.779328    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 21:35:01.891284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:01.890841    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 30 21:35:01.891284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:01.890903    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 30 21:35:01.998029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:01.997754    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d" podUID="3f7a4463-2ca0-4834-a81e-c9eb26bf4392" containerName="payload-processing" containerID="cri-o://b3739d4df6ea7065974c26a2bfbe9eff78c462aa91328a030f08431363222ca2" gracePeriod=25
Oct 30 21:35:02.252225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.252079    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:35:02.255880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.255842    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:35:02.280816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.280754    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 21:35:02.289377 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:35:02.289331    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7a4463_2ca0_4834_a81e_c9eb26bf4392.slice/crio-conmon-b3739d4df6ea7065974c26a2bfbe9eff78c462aa91328a030f08431363222ca2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:35:02.766729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.764428    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 21:35:02.766729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.764511    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 21:35:02.773402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.773366    4633 generic.go:356] "Generic (PLEG): container finished" podID="3f7a4463-2ca0-4834-a81e-c9eb26bf4392" containerID="b3739d4df6ea7065974c26a2bfbe9eff78c462aa91328a030f08431363222ca2" exitCode=0
Oct 30 21:35:02.773556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.773420    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d" event={"ID":"3f7a4463-2ca0-4834-a81e-c9eb26bf4392","Type":"ContainerDied","Data":"b3739d4df6ea7065974c26a2bfbe9eff78c462aa91328a030f08431363222ca2"}
Oct 30 21:35:02.773556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.773478    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d" event={"ID":"3f7a4463-2ca0-4834-a81e-c9eb26bf4392","Type":"ContainerDied","Data":"b1521a239122b54decc3242e4e062843de38e7e88508a18ded04e2ae32d15c9f"}
Oct 30 21:35:02.773556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.773497    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1521a239122b54decc3242e4e062843de38e7e88508a18ded04e2ae32d15c9f"
Oct 30 21:35:02.775908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.775891    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:35:02.890738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.890609    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:35:02.891207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.891146    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:35:02.891951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.891930    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:35:02.988738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.988695    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-kube-api-access-5bb88\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-kube-api-access-5bb88\") pod \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") "
Oct 30 21:35:02.989420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.989396    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-plugins-config-volume\") pod \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") "
Oct 30 21:35:02.990074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.989952    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-plugins-config-volume" pod "3f7a4463-2ca0-4834-a81e-c9eb26bf4392" (UID: "3f7a4463-2ca0-4834-a81e-c9eb26bf4392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:02.990644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.990607    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-routing-overlay-volume" pod "3f7a4463-2ca0-4834-a81e-c9eb26bf4392" (UID: "3f7a4463-2ca0-4834-a81e-c9eb26bf4392"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:02.990803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.990785    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-routing-overlay-volume\") pod \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") "
Oct 30 21:35:02.991179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.991158    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-provider-credentials\") pod \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\" (UID: \"3f7a4463-2ca0-4834-a81e-c9eb26bf4392\") "
Oct 30 21:35:02.992232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.992211    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:02.992402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.992374    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:02.996101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.995993    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-provider-credentials" pod "3f7a4463-2ca0-4834-a81e-c9eb26bf4392" (UID: "3f7a4463-2ca0-4834-a81e-c9eb26bf4392"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:02.996395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:02.996380    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-kube-api-access-5bb88" pod "3f7a4463-2ca0-4834-a81e-c9eb26bf4392" (UID: "3f7a4463-2ca0-4834-a81e-c9eb26bf4392"). InnerVolumeSpecName "kube-api-access-5bb88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:03.095699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.094986    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:03.095699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.095020    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bb88\" (UniqueName: \"kubernetes.io/projected/3f7a4463-2ca0-4834-a81e-c9eb26bf4392-kube-api-access-5bb88\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:03.245141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.245103    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:35:03.245428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.245401    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 21:35:03.761885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.747577    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 30 21:35:03.761885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.747655    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 30 21:35:03.781902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.781866    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"
Oct 30 21:35:03.810304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.810273    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"]
Oct 30 21:35:03.834698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:03.833582    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49d29h2d"]
Oct 30 21:35:04.243113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:04.243072    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:35:04.243443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:04.243412    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 21:35:04.744357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:04.744325    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:35:04.744843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:04.744805    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:35:04.745533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:04.745516    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:35:05.240434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:05.240395    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:35:05.240969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:05.240463    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 21:35:05.794111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:05.794075    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7a4463-2ca0-4834-a81e-c9eb26bf4392" path="/var/lib/kubelet/pods/3f7a4463-2ca0-4834-a81e-c9eb26bf4392/volumes"
Oct 30 21:35:06.255205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:06.255170    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:35:06.255874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:06.255831    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 21:35:07.240142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.240104    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 21:35:07.240413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.240159    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 21:35:07.474113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.474082    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"]
Oct 30 21:35:07.474924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.474908    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3f7a4463-2ca0-4834-a81e-c9eb26bf4392" containerName="payload-processing"
Oct 30 21:35:07.474924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.474924    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7a4463-2ca0-4834-a81e-c9eb26bf4392" containerName="payload-processing"
Oct 30 21:35:07.475055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.475045    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7a4463-2ca0-4834-a81e-c9eb26bf4392" containerName="payload-processing"
Oct 30 21:35:07.480376 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.480359    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.485942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.485918    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-36a099-serving-cert\""
Oct 30 21:35:07.493515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.493494    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"]
Oct 30 21:35:07.648275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.648239    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.648514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.648490    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-metrics-certs\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.648643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.648627    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/475d261a-7603-40c1-8e1a-6e62ee35a5b9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.648733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.648713    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krh2n\" (UniqueName: \"kubernetes.io/projected/475d261a-7603-40c1-8e1a-6e62ee35a5b9-kube-api-access-krh2n\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.756994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.756890    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-metrics-certs\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.758371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.758341    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/475d261a-7603-40c1-8e1a-6e62ee35a5b9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.758826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.758801    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krh2n\" (UniqueName: \"kubernetes.io/projected/475d261a-7603-40c1-8e1a-6e62ee35a5b9-kube-api-access-krh2n\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.763736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.759100    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.770451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.759287    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/475d261a-7603-40c1-8e1a-6e62ee35a5b9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.782764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.782243    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.834437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.834375    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krh2n\" (UniqueName: \"kubernetes.io/projected/475d261a-7603-40c1-8e1a-6e62ee35a5b9-kube-api-access-krh2n\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:07.843963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:07.836463    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-metrics-certs\") pod \"maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:08.106592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:08.106509    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:08.246274 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:08.246224    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:35:08.247387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:08.247229    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:35:08.250910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:08.250887    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:35:08.548643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:08.548602    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"]
Oct 30 21:35:08.977102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:08.977067    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z" event={"ID":"475d261a-7603-40c1-8e1a-6e62ee35a5b9","Type":"ContainerStarted","Data":"9daa6e6a5e8a4299058901b82e2f46f542668fafd68fab877a06fd783ab963f0"}
Oct 30 21:35:09.988840 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:09.988799    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z" event={"ID":"475d261a-7603-40c1-8e1a-6e62ee35a5b9","Type":"ContainerStarted","Data":"5d364f18bd85c35e323cc94959a03af36ee571e8cb00b666ccc18b46e918ff54"}
Oct 30 21:35:09.989309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:09.989182    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:10.012159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:10.012108    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z" podStartSLOduration=2.560055054 podStartE2EDuration="3.012094763s" totalImagesPullingTime="452.039709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:10.009153267 +0000 UTC m=+2018.358880666" watchObservedRunningTime="2026-10-30 21:35:10.012094763 +0000 UTC m=+2018.361822158"
Oct 30 21:35:16.009340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:16.009261    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:35:25.098692 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:35:25.096608    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:35:27.370932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:27.370893    4633 generic.go:356] "Generic (PLEG): container finished" podID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerID="bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b" exitCode=0
Oct 30 21:35:27.375490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:27.372361    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" event={"ID":"3153c1af-b6f5-482d-bcc6-c577c5ad697d","Type":"ContainerDied","Data":"bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b"}
Oct 30 21:35:27.375490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:27.373303    4633 scope.go:122] "RemoveContainer" containerID="bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b"
Oct 30 21:35:27.376868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:27.376804    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:35:27.377012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:27.376915    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:35:27.377012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:27.376970    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:35:28.388832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:28.387559    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" event={"ID":"3153c1af-b6f5-482d-bcc6-c577c5ad697d","Type":"ContainerStarted","Data":"7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33"}
Oct 30 21:35:28.665928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:28.665894    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:35:28.667080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:28.667052    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:35:28.667224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:28.667113    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 21:35:29.430835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:29.430792    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:35:29.666289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:29.666253    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:35:29.666473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:29.666320    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 21:35:30.666455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:30.666423    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:35:30.666919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:30.666502    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 21:35:31.666737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:31.666639    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:35:31.667192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:31.666759    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 21:35:32.666320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:32.666284    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:35:32.666528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:32.666343    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 21:35:33.666574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:33.666532    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:35:33.667096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:33.666614    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 21:35:33.885328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:33.885285    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:35:35.166695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:35.155168    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:35:35.171314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:35.171285    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:35:50.406262 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.406225    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"]
Oct 30 21:35:50.412433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.412411    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.428718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.428636    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"openshift-service-ca.crt\""
Oct 30 21:35:50.429597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.429573    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"kube-root-ca.crt\""
Oct 30 21:35:50.429794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.429572    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"e2e-cmp-legacy-ce1be0-provider-cred\""
Oct 30 21:35:50.429794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.429634    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"routing-overlay\""
Oct 30 21:35:50.429971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.429690    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"payload-processing-external-model-plugins-e2e-extmod-a-729802\""
Oct 30 21:35:50.431802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.431782    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"]
Oct 30 21:35:50.512580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.503088    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.512580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.503153    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.512580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.503188    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.512580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.503244    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2l9\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-kube-api-access-bn2l9\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.561625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.548155    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"payload-processing-external-model-e2e--42f38b37-dockercfg-tbwqf\""
Oct 30 21:35:50.616151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.616111    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.616349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.616192    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.616349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.616243    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.616349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.616315    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2l9\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-kube-api-access-bn2l9\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.621588 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.617583    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.766760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.618442    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.766760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.646301    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:50.766760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:50.650725    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2l9\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-kube-api-access-bn2l9\") pod \"payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:51.054735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.052595    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:51.522352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.522306    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"]
Oct 30 21:35:51.535330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.535301    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.535772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.535512    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"]
Oct 30 21:35:51.556197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.556146    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"e2e-cfl-c82625-provider-cred\""
Oct 30 21:35:51.556569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.556530    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"payload-processing-external-model-e2e--d1dc7551-dockercfg-h5b5m\""
Oct 30 21:35:51.564222 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:35:51.564192    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bab41a_4cb4_42e6_a8c8_d1c582818453.slice/crio-2d77d17b65262a2b89580555e398bfa1c83fc53fa17f9059d950550855aa510a WatchSource:0}: Error finding container 2d77d17b65262a2b89580555e398bfa1c83fc53fa17f9059d950550855aa510a: Status 404 returned error can't find the container with id 2d77d17b65262a2b89580555e398bfa1c83fc53fa17f9059d950550855aa510a
Oct 30 21:35:51.566743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.565982    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"openshift-service-ca.crt\""
Oct 30 21:35:51.566743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.566017    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"kube-root-ca.crt\""
Oct 30 21:35:51.566743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.566115    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a378a2\""
Oct 30 21:35:51.566743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.566123    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"routing-overlay\""
Oct 30 21:35:51.568629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.568605    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"]
Oct 30 21:35:51.581342 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.581314    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.581555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.581531    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7tph\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-kube-api-access-x7tph\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.581708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.581650    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.581794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.581777    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.682758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.682720    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.682918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.682829    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.682970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.682953    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.683082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.683071    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7tph\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-kube-api-access-x7tph\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.683525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.683496    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.683648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.683618    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.685526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.685504    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.707234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.707203    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7tph\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-kube-api-access-x7tph\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:51.862388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:51.862304    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:52.131060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:52.130964    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w" event={"ID":"e5bab41a-4cb4-42e6-a8c8-d1c582818453","Type":"ContainerStarted","Data":"f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f"}
Oct 30 21:35:52.131060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:52.131028    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w" event={"ID":"e5bab41a-4cb4-42e6-a8c8-d1c582818453","Type":"ContainerStarted","Data":"2d77d17b65262a2b89580555e398bfa1c83fc53fa17f9059d950550855aa510a"}
Oct 30 21:35:52.131060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:52.131053    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:35:52.172613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:52.172585    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"]
Oct 30 21:35:52.174485 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:35:52.174460    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ce8b31_f35b_4b77_8afc_618429756c6b.slice/crio-d6083b63e5fb7e6c537a5b1244fbd541811ceb711563c7a8f785c441ed59cc48 WatchSource:0}: Error finding container d6083b63e5fb7e6c537a5b1244fbd541811ceb711563c7a8f785c441ed59cc48: Status 404 returned error can't find the container with id d6083b63e5fb7e6c537a5b1244fbd541811ceb711563c7a8f785c441ed59cc48
Oct 30 21:35:52.188927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:52.188862    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w" podStartSLOduration=2.188844561 podStartE2EDuration="2.188844561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:52.17774431 +0000 UTC m=+2060.527471708" watchObservedRunningTime="2026-10-30 21:35:52.188844561 +0000 UTC m=+2060.538571958"
Oct 30 21:35:53.144275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:53.144233    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb" event={"ID":"67ce8b31-f35b-4b77-8afc-618429756c6b","Type":"ContainerStarted","Data":"22ab701427317833e9e4b1333ec6fe3b96efcf89f42234435d6122d9c91505ca"}
Oct 30 21:35:53.144949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:53.144307    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb" event={"ID":"67ce8b31-f35b-4b77-8afc-618429756c6b","Type":"ContainerStarted","Data":"d6083b63e5fb7e6c537a5b1244fbd541811ceb711563c7a8f785c441ed59cc48"}
Oct 30 21:35:53.144949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:53.144608    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:35:53.171240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:53.171174    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb" podStartSLOduration=2.171156884 podStartE2EDuration="2.171156884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:53.166401391 +0000 UTC m=+2061.516128803" watchObservedRunningTime="2026-10-30 21:35:53.171156884 +0000 UTC m=+2061.520884293"
Oct 30 21:35:55.019236 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:35:55.019188    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:35:55.923625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.923588    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"]
Oct 30 21:35:55.944325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.944287    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:55.946828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.946795    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"]
Oct 30 21:35:55.954114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.954090    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"openshift-service-ca.crt\""
Oct 30 21:35:55.954449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.954357    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"routing-overlay\""
Oct 30 21:35:55.954620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.954393    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1ed97\""
Oct 30 21:35:55.954778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.954398    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"kube-root-ca.crt\""
Oct 30 21:35:55.954913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.954870    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-c07a671227da\""
Oct 30 21:35:55.954913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.954897    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-493f269f193a\""
Oct 30 21:35:55.955040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:55.954997    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-e2e--650b641a-dockercfg-s6bl5\""
Oct 30 21:35:56.020185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.020155    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.020570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.020365    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.020570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.020412    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsnpw\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-kube-api-access-vsnpw\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.020570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.020495    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.121748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.121706    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.121918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.121774    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnpw\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-kube-api-access-vsnpw\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.121918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.121823    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.122074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.121950    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.122467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.122435    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.122729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.122649    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.124989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.124964    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.136345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.136324    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnpw\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-kube-api-access-vsnpw\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.271183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.271100    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:56.670347 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:35:56.670319    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54911aee_0993_4ffd_8871_320cc995be75.slice/crio-9ff579cd409db00585e1cf46096e44ee152b4b465dac494e5ce6fd7bc28adc76 WatchSource:0}: Error finding container 9ff579cd409db00585e1cf46096e44ee152b4b465dac494e5ce6fd7bc28adc76: Status 404 returned error can't find the container with id 9ff579cd409db00585e1cf46096e44ee152b4b465dac494e5ce6fd7bc28adc76
Oct 30 21:35:56.671232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:56.671208    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"]
Oct 30 21:35:57.194090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:57.194051    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6" event={"ID":"54911aee-0993-4ffd-8871-320cc995be75","Type":"ContainerStarted","Data":"242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b"}
Oct 30 21:35:57.194512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:57.194120    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6" event={"ID":"54911aee-0993-4ffd-8871-320cc995be75","Type":"ContainerStarted","Data":"9ff579cd409db00585e1cf46096e44ee152b4b465dac494e5ce6fd7bc28adc76"}
Oct 30 21:35:57.194512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:57.194148    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:35:57.215362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:35:57.215299    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6" podStartSLOduration=2.215286028 podStartE2EDuration="2.215286028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:57.211928095 +0000 UTC m=+2065.561655504" watchObservedRunningTime="2026-10-30 21:35:57.215286028 +0000 UTC m=+2065.565013424"
Oct 30 21:36:03.147956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:03.147922    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:36:04.157381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:04.157348    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:36:08.207655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:08.207625    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:36:25.015723 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:36:25.015655    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:36:32.629566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:32.629065    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:36:32.637693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:32.637653    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:36:32.647896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:32.647873    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:36:32.655196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:32.655172    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:36:42.913755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:42.910840    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:36:50.160101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.160070    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"]
Oct 30 21:36:50.166467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.166449    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.203801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.203763    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-fb2b58-openshift-default-dockercfg-7jgd8\""
Oct 30 21:36:50.211278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.211248    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"]
Oct 30 21:36:50.299062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299023    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.299237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299183    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.299289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299249    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.299375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299339    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.299560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299540    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8psz\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-kube-api-access-n8psz\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.299624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299609    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdd9f9d-6b83-495e-804f-e85d4185d396-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.299687 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299653    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.299860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299839    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.299954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.299926    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402011    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402077    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402173    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402260    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402315    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402344    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402408    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402539    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8psz\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-kube-api-access-n8psz\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402614    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402628    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402623    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdd9f9d-6b83-495e-804f-e85d4185d396-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402716    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.402816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.402754    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.403545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.403522    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdd9f9d-6b83-495e-804f-e85d4185d396-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.404902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.404881    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.405021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.404997    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.412846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.412793    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.417426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.417401    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8psz\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-kube-api-access-n8psz\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.482554 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.482523    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:50.791080 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:36:50.791048    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbdd9f9d_6b83_495e_804f_e85d4185d396.slice/crio-d24df9048c2e11645f203c5de68f2b363820eb6f2df1b791ea3ff7546c16ff0b WatchSource:0}: Error finding container d24df9048c2e11645f203c5de68f2b363820eb6f2df1b791ea3ff7546c16ff0b: Status 404 returned error can't find the container with id d24df9048c2e11645f203c5de68f2b363820eb6f2df1b791ea3ff7546c16ff0b
Oct 30 21:36:50.791959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.791935    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"]
Oct 30 21:36:50.794408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.794369    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:36:50.794484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.794456    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:36:50.794523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:50.794501    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:36:51.061742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:51.061479    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" event={"ID":"bbdd9f9d-6b83-495e-804f-e85d4185d396","Type":"ContainerStarted","Data":"0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164"}
Oct 30 21:36:51.061742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:51.061543    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" event={"ID":"bbdd9f9d-6b83-495e-804f-e85d4185d396","Type":"ContainerStarted","Data":"d24df9048c2e11645f203c5de68f2b363820eb6f2df1b791ea3ff7546c16ff0b"}
Oct 30 21:36:51.099756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:51.096354    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podStartSLOduration=1.09633494 podStartE2EDuration="1.09633494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:51.093828814 +0000 UTC m=+2119.443556232" watchObservedRunningTime="2026-10-30 21:36:51.09633494 +0000 UTC m=+2119.446062337"
Oct 30 21:36:51.483114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:51.483080    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:51.484649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:51.484624    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:36:51.484835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:51.484714    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 21:36:52.483286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:52.483252    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:36:52.483783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:52.483316    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 21:36:53.483165 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:53.483126    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:36:53.483348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:53.483181    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 21:36:54.482942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:54.482902    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:36:54.483125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:54.482958    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 21:36:55.042736 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:36:55.042653    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:36:55.486207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:55.486166    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:36:55.486542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:55.486509    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 21:36:56.483397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:56.483355    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:36:56.483942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:56.483438    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 21:36:57.483284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:57.483246    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:36:57.483493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:57.483312    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 30 21:36:58.494979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:58.494931    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:58.495824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:58.495796    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:36:58.496755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:36:58.496659    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:37:22.246040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.246007    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"]
Oct 30 21:37:22.255251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.255211    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.262765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.261985    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-dabf56f8ec6a\""
Oct 30 21:37:22.262765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.262712    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-2f614df36ac6\""
Oct 30 21:37:22.263276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.263046    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"openshift-service-ca.crt\""
Oct 30 21:37:22.263276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.263134    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"kube-root-ca.crt\""
Oct 30 21:37:22.263276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.263174    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-plugins-e2e-extmod-b-36a099\""
Oct 30 21:37:22.263276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.263206    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-e2e--3e617b48-dockercfg-d4sw2\""
Oct 30 21:37:22.263276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.263246    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"routing-overlay\""
Oct 30 21:37:22.269030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.264237    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"]
Oct 30 21:37:22.351825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.351780    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.352107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.351869    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.352107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.352063    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5g9\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-kube-api-access-vk5g9\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.352260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.352148    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.499754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.499214    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.499754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.499335    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.499754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.499623    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5g9\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-kube-api-access-vk5g9\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.499754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.499687    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.524188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.524133    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.552695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.551369    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.586808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.581145    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5g9\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-kube-api-access-vk5g9\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.615719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.613126    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:22.620241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:22.620197    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:23.336210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:23.336170    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"]
Oct 30 21:37:23.994737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:23.990243    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" event={"ID":"44d1d842-5432-440d-92b9-8fd262ee7292","Type":"ContainerStarted","Data":"1fbff80809cbcc0153cf4bc27fb79c74a82063a904a21d09cddf486425a1e218"}
Oct 30 21:37:23.994737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:23.990312    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" event={"ID":"44d1d842-5432-440d-92b9-8fd262ee7292","Type":"ContainerStarted","Data":"95ac65312270d2d06993be426edde6cafd5e8ce555c1544011d0ac2559630b82"}
Oct 30 21:37:23.994737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:23.991805    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:24.014096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:24.014027    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" podStartSLOduration=2.014008167 podStartE2EDuration="2.014008167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:24.013552027 +0000 UTC m=+2152.363279425" watchObservedRunningTime="2026-10-30 21:37:24.014008167 +0000 UTC m=+2152.363735565"
Oct 30 21:37:24.132638 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:24.132607    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-36a099/agc-cred-dabf56f8ec6a: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:24.132638 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:24.132633    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:24.132959 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:24.132719    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials podName:44d1d842-5432-440d-92b9-8fd262ee7292 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:24.632699567 +0000 UTC m=+2152.982426957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292") : secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:24.723720 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:24.702104    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-36a099/agc-cred-dabf56f8ec6a: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:24.723720 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:24.702149    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:24.723720 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:24.702240    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials podName:44d1d842-5432-440d-92b9-8fd262ee7292 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:25.702216908 +0000 UTC m=+2154.051944303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292") : secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:25.114300 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:25.114215    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:37:25.226273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:25.226239    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"]
Oct 30 21:37:25.804659 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:25.804627    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-36a099/agc-cred-dabf56f8ec6a: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:25.807815 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:25.807794    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:25.808037 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:25.808021    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials podName:44d1d842-5432-440d-92b9-8fd262ee7292 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:27.807998602 +0000 UTC m=+2156.157725995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292") : secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:27.847004 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:27.846972    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-36a099/agc-cred-dabf56f8ec6a: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:27.847004 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:27.846998    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:27.847454 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:27.847072    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials podName:44d1d842-5432-440d-92b9-8fd262ee7292 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:31.847053221 +0000 UTC m=+2160.196780634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292") : secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:31.950942 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:31.950908    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-36a099/agc-cred-dabf56f8ec6a: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:31.950942 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:31.950942    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z: secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:31.951368 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:31.951003    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials podName:44d1d842-5432-440d-92b9-8fd262ee7292 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:39.950984989 +0000 UTC m=+2168.300712366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292") : secret "agc-cred-dabf56f8ec6a" not found
Oct 30 21:37:32.074642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.074581    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" podUID="44d1d842-5432-440d-92b9-8fd262ee7292" containerName="payload-processing" containerID="cri-o://1fbff80809cbcc0153cf4bc27fb79c74a82063a904a21d09cddf486425a1e218" gracePeriod=25
Oct 30 21:37:32.246382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.246288    4633 generic.go:356] "Generic (PLEG): container finished" podID="44d1d842-5432-440d-92b9-8fd262ee7292" containerID="1fbff80809cbcc0153cf4bc27fb79c74a82063a904a21d09cddf486425a1e218" exitCode=0
Oct 30 21:37:32.246382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.246357    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" event={"ID":"44d1d842-5432-440d-92b9-8fd262ee7292","Type":"ContainerDied","Data":"1fbff80809cbcc0153cf4bc27fb79c74a82063a904a21d09cddf486425a1e218"}
Oct 30 21:37:32.262740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.257860    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"]
Oct 30 21:37:32.438388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.438356    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:32.732486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.732450    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-kube-api-access-vk5g9\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-kube-api-access-vk5g9\") pod \"44d1d842-5432-440d-92b9-8fd262ee7292\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") "
Oct 30 21:37:32.732848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.732823    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-routing-overlay-volume\") pod \"44d1d842-5432-440d-92b9-8fd262ee7292\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") "
Oct 30 21:37:32.733045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.733026    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials\") pod \"44d1d842-5432-440d-92b9-8fd262ee7292\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") "
Oct 30 21:37:32.733437 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.733416    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-plugins-config-volume\") pod \"44d1d842-5432-440d-92b9-8fd262ee7292\" (UID: \"44d1d842-5432-440d-92b9-8fd262ee7292\") "
Oct 30 21:37:32.781500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.735588    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-routing-overlay-volume" pod "44d1d842-5432-440d-92b9-8fd262ee7292" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:32.782071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.782032    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-plugins-config-volume" pod "44d1d842-5432-440d-92b9-8fd262ee7292" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:32.865703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.853203    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:32.865703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.853242    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d1d842-5432-440d-92b9-8fd262ee7292-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:32.865703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.860321    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-kube-api-access-vk5g9" pod "44d1d842-5432-440d-92b9-8fd262ee7292" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292"). InnerVolumeSpecName "kube-api-access-vk5g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:32.944137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:32.890115    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials" pod "44d1d842-5432-440d-92b9-8fd262ee7292" (UID: "44d1d842-5432-440d-92b9-8fd262ee7292"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:33.023253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:33.020082    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk5g9\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-kube-api-access-vk5g9\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:33.023253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:33.020118    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/44d1d842-5432-440d-92b9-8fd262ee7292-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:33.257197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:33.257157    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z" event={"ID":"44d1d842-5432-440d-92b9-8fd262ee7292","Type":"ContainerDied","Data":"95ac65312270d2d06993be426edde6cafd5e8ce555c1544011d0ac2559630b82"}
Oct 30 21:37:33.257197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:33.257195    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"
Oct 30 21:37:33.257438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:33.257245    4633 scope.go:122] "RemoveContainer" containerID="1fbff80809cbcc0153cf4bc27fb79c74a82063a904a21d09cddf486425a1e218"
Oct 30 21:37:33.287102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:33.287020    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"]
Oct 30 21:37:33.293033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:33.293001    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-5848j9t4z"]
Oct 30 21:37:33.808604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:33.808573    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d1d842-5432-440d-92b9-8fd262ee7292" path="/var/lib/kubelet/pods/44d1d842-5432-440d-92b9-8fd262ee7292/volumes"
Oct 30 21:37:35.683650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.683610    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"]
Oct 30 21:37:35.684575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.684555    4633 cpu_manager.go:414] "containerMap: removing container" podUID="44d1d842-5432-440d-92b9-8fd262ee7292" containerName="payload-processing"
Oct 30 21:37:35.684634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.684580    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d1d842-5432-440d-92b9-8fd262ee7292" containerName="payload-processing"
Oct 30 21:37:35.684744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.684734    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d1d842-5432-440d-92b9-8fd262ee7292" containerName="payload-processing"
Oct 30 21:37:35.690543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.690521    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.705456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.705433    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"agc-cred-b22596d4938d\""
Oct 30 21:37:35.705603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.705453    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"payload-processing-external-model-e2e--42f38b37-dockercfg-6wsvt\""
Oct 30 21:37:35.705603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.705470    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.705777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.705756    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl69t\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-kube-api-access-rl69t\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.705845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.705825    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.706051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.706022    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.711651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.711626    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"]
Oct 30 21:37:35.853698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.853007    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.853698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.853300    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl69t\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-kube-api-access-rl69t\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.853698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.853371    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.853698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.853579    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.869697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.866413    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.893695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.888718    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.994743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.988731    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl69t\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-kube-api-access-rl69t\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:35.994743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:35.989236    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:36.012699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:36.009044    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:36.472039 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:37:36.472013    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2891432_14a1_4a32_870e_4af50b14d10f.slice/crio-801aa8f91f57d16e8932c895811bc088b9888670449f95e1622c4cec0a8bc55d WatchSource:0}: Error finding container 801aa8f91f57d16e8932c895811bc088b9888670449f95e1622c4cec0a8bc55d: Status 404 returned error can't find the container with id 801aa8f91f57d16e8932c895811bc088b9888670449f95e1622c4cec0a8bc55d
Oct 30 21:37:36.475709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:36.474634    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"]
Oct 30 21:37:37.258739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.258677    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w" podUID="e5bab41a-4cb4-42e6-a8c8-d1c582818453" containerName="payload-processing" containerID="cri-o://f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f" gracePeriod=25
Oct 30 21:37:37.321302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.321257    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5" event={"ID":"e2891432-14a1-4a32-870e-4af50b14d10f","Type":"ContainerStarted","Data":"bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03"}
Oct 30 21:37:37.321472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.321316    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5" event={"ID":"e2891432-14a1-4a32-870e-4af50b14d10f","Type":"ContainerStarted","Data":"801aa8f91f57d16e8932c895811bc088b9888670449f95e1622c4cec0a8bc55d"}
Oct 30 21:37:37.321472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.321336    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:37.341934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.341879    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5" podStartSLOduration=2.341864875 podStartE2EDuration="2.341864875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:37.340097745 +0000 UTC m=+2165.689825142" watchObservedRunningTime="2026-10-30 21:37:37.341864875 +0000 UTC m=+2165.691592271"
Oct 30 21:37:37.580815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.580787    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:37:37.738866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.738836    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-kube-api-access-bn2l9\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-kube-api-access-bn2l9\") pod \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") "
Oct 30 21:37:37.739113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.738903    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-provider-credentials\") pod \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") "
Oct 30 21:37:37.739113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.739095    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-plugins-config-volume\") pod \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") "
Oct 30 21:37:37.739264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.739156    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-routing-overlay-volume\") pod \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\" (UID: \"e5bab41a-4cb4-42e6-a8c8-d1c582818453\") "
Oct 30 21:37:37.739526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.739503    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-routing-overlay-volume" pod "e5bab41a-4cb4-42e6-a8c8-d1c582818453" (UID: "e5bab41a-4cb4-42e6-a8c8-d1c582818453"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:37.739619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.739561    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-plugins-config-volume" pod "e5bab41a-4cb4-42e6-a8c8-d1c582818453" (UID: "e5bab41a-4cb4-42e6-a8c8-d1c582818453"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:37.740921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.740899    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.741024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.740928    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5bab41a-4cb4-42e6-a8c8-d1c582818453-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.741731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.741695    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-kube-api-access-bn2l9" pod "e5bab41a-4cb4-42e6-a8c8-d1c582818453" (UID: "e5bab41a-4cb4-42e6-a8c8-d1c582818453"). InnerVolumeSpecName "kube-api-access-bn2l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:37.741972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.741945    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-provider-credentials" pod "e5bab41a-4cb4-42e6-a8c8-d1c582818453" (UID: "e5bab41a-4cb4-42e6-a8c8-d1c582818453"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:37.861692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.858621    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2l9\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-kube-api-access-bn2l9\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.861692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:37.858660    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5bab41a-4cb4-42e6-a8c8-d1c582818453-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:38.388564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.388515    4633 generic.go:356] "Generic (PLEG): container finished" podID="e5bab41a-4cb4-42e6-a8c8-d1c582818453" containerID="f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f" exitCode=0
Oct 30 21:37:38.390962 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.389194    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"
Oct 30 21:37:38.391304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.389269    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w" event={"ID":"e5bab41a-4cb4-42e6-a8c8-d1c582818453","Type":"ContainerDied","Data":"f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f"}
Oct 30 21:37:38.391500 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.391479    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w" event={"ID":"e5bab41a-4cb4-42e6-a8c8-d1c582818453","Type":"ContainerDied","Data":"2d77d17b65262a2b89580555e398bfa1c83fc53fa17f9059d950550855aa510a"}
Oct 30 21:37:38.391617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.391592    4633 scope.go:122] "RemoveContainer" containerID="f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f"
Oct 30 21:37:38.446896 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.446860    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"]
Oct 30 21:37:38.458535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.458494    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-5d5brqj2w"]
Oct 30 21:37:38.462310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.462281    4633 scope.go:122] "RemoveContainer" containerID="f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f"
Oct 30 21:37:38.463358 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:38.463199    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f\": container with ID starting with f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f not found: ID does not exist" containerID="f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f"
Oct 30 21:37:38.463358 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.463264    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f"} err="failed to get container status \"f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f\": rpc error: code = NotFound desc = could not find container \"f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f\": container with ID starting with f14790602e29770007876e7e0ff56d07117a33c3968089c85080b4fb2289b04f not found: ID does not exist"
Oct 30 21:37:38.593133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.593092    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"]
Oct 30 21:37:38.797214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.797184    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"]
Oct 30 21:37:38.798050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.798033    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e5bab41a-4cb4-42e6-a8c8-d1c582818453" containerName="payload-processing"
Oct 30 21:37:38.798110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.798052    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bab41a-4cb4-42e6-a8c8-d1c582818453" containerName="payload-processing"
Oct 30 21:37:38.798208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.798199    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bab41a-4cb4-42e6-a8c8-d1c582818453" containerName="payload-processing"
Oct 30 21:37:38.803445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.803421    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:38.816927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.816760    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"payload-processing-external-model-plugins-e2e-extmod-a-fb2b58\""
Oct 30 21:37:38.816927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.816823    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"e2e-cmp-legacy-83bfe6-provider-cred\""
Oct 30 21:37:38.816927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.816891    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"openshift-service-ca.crt\""
Oct 30 21:37:38.816927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.816893    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"kube-root-ca.crt\""
Oct 30 21:37:38.817383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.817363    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"routing-overlay\""
Oct 30 21:37:38.820193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.820167    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"]
Oct 30 21:37:38.899390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.899354    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kls\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-kube-api-access-f7kls\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:38.899390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.899395    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:38.899625 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.899533    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:38.899765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:38.899738    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.001534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.001485    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.001774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.001720    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.001909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.001890    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kls\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-kube-api-access-f7kls\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.001976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.001930    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.003189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.002727    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.003189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.002923    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.014244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.014188    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.017707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.017639    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kls\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-kube-api-access-f7kls\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.122852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.122734    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:39.849384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.848141    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bab41a-4cb4-42e6-a8c8-d1c582818453" path="/var/lib/kubelet/pods/e5bab41a-4cb4-42e6-a8c8-d1c582818453/volumes"
Oct 30 21:37:39.849384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.848705    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:37:39.954640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:39.954610    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"]
Oct 30 21:37:39.960477 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:37:39.960440    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7dc822_4903_4a99_89cc_799a73549412.slice/crio-605cef4d3ffe53d7171ad1bc99d6227071fec3fa037fbe80af81bde61c75b41d WatchSource:0}: Error finding container 605cef4d3ffe53d7171ad1bc99d6227071fec3fa037fbe80af81bde61c75b41d: Status 404 returned error can't find the container with id 605cef4d3ffe53d7171ad1bc99d6227071fec3fa037fbe80af81bde61c75b41d
Oct 30 21:37:40.519174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:40.519135    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr" event={"ID":"7c7dc822-4903-4a99-89cc-799a73549412","Type":"ContainerStarted","Data":"48e6406cd4e0ee2bcf10cc126b5b7a8e23802f5f52647594e788425dec5d46d7"}
Oct 30 21:37:40.519354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:40.519199    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr" event={"ID":"7c7dc822-4903-4a99-89cc-799a73549412","Type":"ContainerStarted","Data":"605cef4d3ffe53d7171ad1bc99d6227071fec3fa037fbe80af81bde61c75b41d"}
Oct 30 21:37:40.519354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:40.519230    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:40.543132 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:40.543072    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr" podStartSLOduration=2.543055831 podStartE2EDuration="2.543055831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:40.540818675 +0000 UTC m=+2168.890546072" watchObservedRunningTime="2026-10-30 21:37:40.543055831 +0000 UTC m=+2168.892783229"
Oct 30 21:37:43.593963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:43.593918    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb" podUID="67ce8b31-f35b-4b77-8afc-618429756c6b" containerName="payload-processing" containerID="cri-o://22ab701427317833e9e4b1333ec6fe3b96efcf89f42234435d6122d9c91505ca" gracePeriod=25
Oct 30 21:37:43.756523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:43.756492    4633 generic.go:356] "Generic (PLEG): container finished" podID="67ce8b31-f35b-4b77-8afc-618429756c6b" containerID="22ab701427317833e9e4b1333ec6fe3b96efcf89f42234435d6122d9c91505ca" exitCode=0
Oct 30 21:37:43.756720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:43.756565    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb" event={"ID":"67ce8b31-f35b-4b77-8afc-618429756c6b","Type":"ContainerDied","Data":"22ab701427317833e9e4b1333ec6fe3b96efcf89f42234435d6122d9c91505ca"}
Oct 30 21:37:43.903044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:43.903017    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:37:44.046632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.046598    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-plugins-config-volume\") pod \"67ce8b31-f35b-4b77-8afc-618429756c6b\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") "
Oct 30 21:37:44.047143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.046842    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-provider-credentials\") pod \"67ce8b31-f35b-4b77-8afc-618429756c6b\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") "
Oct 30 21:37:44.047143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.046910    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-routing-overlay-volume\") pod \"67ce8b31-f35b-4b77-8afc-618429756c6b\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") "
Oct 30 21:37:44.047143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.047102    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-kube-api-access-x7tph\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-kube-api-access-x7tph\") pod \"67ce8b31-f35b-4b77-8afc-618429756c6b\" (UID: \"67ce8b31-f35b-4b77-8afc-618429756c6b\") "
Oct 30 21:37:44.047923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.047891    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-plugins-config-volume" pod "67ce8b31-f35b-4b77-8afc-618429756c6b" (UID: "67ce8b31-f35b-4b77-8afc-618429756c6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:44.048248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.048228    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:44.049181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.048689    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-routing-overlay-volume" pod "67ce8b31-f35b-4b77-8afc-618429756c6b" (UID: "67ce8b31-f35b-4b77-8afc-618429756c6b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:44.051409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.051373    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-provider-credentials" pod "67ce8b31-f35b-4b77-8afc-618429756c6b" (UID: "67ce8b31-f35b-4b77-8afc-618429756c6b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:44.054069 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.054045    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-kube-api-access-x7tph" pod "67ce8b31-f35b-4b77-8afc-618429756c6b" (UID: "67ce8b31-f35b-4b77-8afc-618429756c6b"). InnerVolumeSpecName "kube-api-access-x7tph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:44.149888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.149804    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:44.149888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.149837    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67ce8b31-f35b-4b77-8afc-618429756c6b-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:44.149888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.149851    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7tph\" (UniqueName: \"kubernetes.io/projected/67ce8b31-f35b-4b77-8afc-618429756c6b-kube-api-access-x7tph\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:44.767489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.767463    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"
Oct 30 21:37:44.767489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.767473    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb" event={"ID":"67ce8b31-f35b-4b77-8afc-618429756c6b","Type":"ContainerDied","Data":"d6083b63e5fb7e6c537a5b1244fbd541811ceb711563c7a8f785c441ed59cc48"}
Oct 30 21:37:44.768017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.767531    4633 scope.go:122] "RemoveContainer" containerID="22ab701427317833e9e4b1333ec6fe3b96efcf89f42234435d6122d9c91505ca"
Oct 30 21:37:44.797342 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.797315    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"]
Oct 30 21:37:44.815046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:44.815004    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbnbrkb"]
Oct 30 21:37:45.793140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:45.793100    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ce8b31-f35b-4b77-8afc-618429756c6b" path="/var/lib/kubelet/pods/67ce8b31-f35b-4b77-8afc-618429756c6b/volumes"
Oct 30 21:37:48.393589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:48.393552    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"]
Oct 30 21:37:48.394928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:48.394905    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:51.741316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:51.741285    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:37:53.398595 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.395758    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5" podUID="e2891432-14a1-4a32-870e-4af50b14d10f" containerName="payload-processing" containerID="cri-o://bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03" gracePeriod=25
Oct 30 21:37:53.704189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.704162    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:53.782570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.782531    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-kube-api-access-rl69t\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-kube-api-access-rl69t\") pod \"e2891432-14a1-4a32-870e-4af50b14d10f\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") "
Oct 30 21:37:53.782865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.782844    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-routing-overlay-volume\") pod \"e2891432-14a1-4a32-870e-4af50b14d10f\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") "
Oct 30 21:37:53.782978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.782883    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-plugins-config-volume\") pod \"e2891432-14a1-4a32-870e-4af50b14d10f\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") "
Oct 30 21:37:53.783180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.783159    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-provider-credentials\") pod \"e2891432-14a1-4a32-870e-4af50b14d10f\" (UID: \"e2891432-14a1-4a32-870e-4af50b14d10f\") "
Oct 30 21:37:53.783403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.783370    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-plugins-config-volume" pod "e2891432-14a1-4a32-870e-4af50b14d10f" (UID: "e2891432-14a1-4a32-870e-4af50b14d10f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:53.783490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.783396    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-routing-overlay-volume" pod "e2891432-14a1-4a32-870e-4af50b14d10f" (UID: "e2891432-14a1-4a32-870e-4af50b14d10f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:53.784412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.784383    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:53.784521 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.784416    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2891432-14a1-4a32-870e-4af50b14d10f-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:53.785807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.785780    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-kube-api-access-rl69t" pod "e2891432-14a1-4a32-870e-4af50b14d10f" (UID: "e2891432-14a1-4a32-870e-4af50b14d10f"). InnerVolumeSpecName "kube-api-access-rl69t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:53.786038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.786017    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-provider-credentials" pod "e2891432-14a1-4a32-870e-4af50b14d10f" (UID: "e2891432-14a1-4a32-870e-4af50b14d10f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:53.857031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.856995    4633 generic.go:356] "Generic (PLEG): container finished" podID="e2891432-14a1-4a32-870e-4af50b14d10f" containerID="bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03" exitCode=0
Oct 30 21:37:53.857209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.857035    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5" event={"ID":"e2891432-14a1-4a32-870e-4af50b14d10f","Type":"ContainerDied","Data":"bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03"}
Oct 30 21:37:53.857209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.857092    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5" event={"ID":"e2891432-14a1-4a32-870e-4af50b14d10f","Type":"ContainerDied","Data":"801aa8f91f57d16e8932c895811bc088b9888670449f95e1622c4cec0a8bc55d"}
Oct 30 21:37:53.857209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.857117    4633 scope.go:122] "RemoveContainer" containerID="bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03"
Oct 30 21:37:53.857209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.857051    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"
Oct 30 21:37:53.869024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.869002    4633 scope.go:122] "RemoveContainer" containerID="bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03"
Oct 30 21:37:53.869344 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:53.869323    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03\": container with ID starting with bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03 not found: ID does not exist" containerID="bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03"
Oct 30 21:37:53.869417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.869356    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03"} err="failed to get container status \"bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03\": rpc error: code = NotFound desc = could not find container \"bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03\": container with ID starting with bab2e7b77bd31e99e329c5e455f0ef64a6b41ae94dee293e00541a60feef1f03 not found: ID does not exist"
Oct 30 21:37:53.878203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.878174    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"]
Oct 30 21:37:53.883757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.883728    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69ccbnzl5"]
Oct 30 21:37:53.886616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.886592    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl69t\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-kube-api-access-rl69t\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:53.886616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:53.886618    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2891432-14a1-4a32-870e-4af50b14d10f-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.297571 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:54.291198    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"]
Oct 30 21:37:55.029079 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:55.029037    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:37:55.796259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:55.796225    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2891432-14a1-4a32-870e-4af50b14d10f" path="/var/lib/kubelet/pods/e2891432-14a1-4a32-870e-4af50b14d10f/volumes"
Oct 30 21:37:59.291856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.291806    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6" podUID="54911aee-0993-4ffd-8871-320cc995be75" containerName="payload-processing" containerID="cri-o://242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b" gracePeriod=25
Oct 30 21:37:59.623722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.623690    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:37:59.673754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.673720    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-routing-overlay-volume\") pod \"54911aee-0993-4ffd-8871-320cc995be75\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") "
Oct 30 21:37:59.673754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.673759    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-provider-credentials\") pod \"54911aee-0993-4ffd-8871-320cc995be75\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") "
Oct 30 21:37:59.673970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.673812    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-plugins-config-volume\") pod \"54911aee-0993-4ffd-8871-320cc995be75\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") "
Oct 30 21:37:59.673970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.673866    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-kube-api-access-vsnpw\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-kube-api-access-vsnpw\") pod \"54911aee-0993-4ffd-8871-320cc995be75\" (UID: \"54911aee-0993-4ffd-8871-320cc995be75\") "
Oct 30 21:37:59.674218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.674190    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-routing-overlay-volume" pod "54911aee-0993-4ffd-8871-320cc995be75" (UID: "54911aee-0993-4ffd-8871-320cc995be75"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:59.674313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.674287    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-plugins-config-volume" pod "54911aee-0993-4ffd-8871-320cc995be75" (UID: "54911aee-0993-4ffd-8871-320cc995be75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:59.674773 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.674751    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:59.674839 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.674779    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54911aee-0993-4ffd-8871-320cc995be75-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:59.676290 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.676251    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-provider-credentials" pod "54911aee-0993-4ffd-8871-320cc995be75" (UID: "54911aee-0993-4ffd-8871-320cc995be75"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:59.676473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.676442    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-kube-api-access-vsnpw" pod "54911aee-0993-4ffd-8871-320cc995be75" (UID: "54911aee-0993-4ffd-8871-320cc995be75"). InnerVolumeSpecName "kube-api-access-vsnpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:59.776989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.776953    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:59.776989 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.776985    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsnpw\" (UniqueName: \"kubernetes.io/projected/54911aee-0993-4ffd-8871-320cc995be75-kube-api-access-vsnpw\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:59.970409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.970373    4633 generic.go:356] "Generic (PLEG): container finished" podID="54911aee-0993-4ffd-8871-320cc995be75" containerID="242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b" exitCode=0
Oct 30 21:37:59.970600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.970423    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"
Oct 30 21:37:59.970600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.970464    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6" event={"ID":"54911aee-0993-4ffd-8871-320cc995be75","Type":"ContainerDied","Data":"242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b"}
Oct 30 21:37:59.970600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.970523    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6" event={"ID":"54911aee-0993-4ffd-8871-320cc995be75","Type":"ContainerDied","Data":"9ff579cd409db00585e1cf46096e44ee152b4b465dac494e5ce6fd7bc28adc76"}
Oct 30 21:37:59.970600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.970545    4633 scope.go:122] "RemoveContainer" containerID="242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b"
Oct 30 21:37:59.983092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.983068    4633 scope.go:122] "RemoveContainer" containerID="242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b"
Oct 30 21:37:59.983401 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:37:59.983385    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b\": container with ID starting with 242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b not found: ID does not exist" containerID="242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b"
Oct 30 21:37:59.983447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.983410    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b"} err="failed to get container status \"242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b\": rpc error: code = NotFound desc = could not find container \"242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b\": container with ID starting with 242683615bd3c20bec84e2ad3736df3c3c7aed6be3abd7e7c064095560630d3b not found: ID does not exist"
Oct 30 21:37:59.995060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.995029    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"]
Oct 30 21:37:59.999925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:37:59.999895    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-9fb7sbzc6"]
Oct 30 21:38:01.794923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:01.794892    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54911aee-0993-4ffd-8871-320cc995be75" path="/var/lib/kubelet/pods/54911aee-0993-4ffd-8871-320cc995be75/volumes"
Oct 30 21:38:06.729593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.729562    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"]
Oct 30 21:38:06.730635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730613    4633 cpu_manager.go:414] "containerMap: removing container" podUID="67ce8b31-f35b-4b77-8afc-618429756c6b" containerName="payload-processing"
Oct 30 21:38:06.730635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730633    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ce8b31-f35b-4b77-8afc-618429756c6b" containerName="payload-processing"
Oct 30 21:38:06.730848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730649    4633 cpu_manager.go:414] "containerMap: removing container" podUID="54911aee-0993-4ffd-8871-320cc995be75" containerName="payload-processing"
Oct 30 21:38:06.730848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730659    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54911aee-0993-4ffd-8871-320cc995be75" containerName="payload-processing"
Oct 30 21:38:06.730848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730770    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e2891432-14a1-4a32-870e-4af50b14d10f" containerName="payload-processing"
Oct 30 21:38:06.730848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730780    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2891432-14a1-4a32-870e-4af50b14d10f" containerName="payload-processing"
Oct 30 21:38:06.731074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730930    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ce8b31-f35b-4b77-8afc-618429756c6b" containerName="payload-processing"
Oct 30 21:38:06.731074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730941    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="54911aee-0993-4ffd-8871-320cc995be75" containerName="payload-processing"
Oct 30 21:38:06.731074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.730961    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2891432-14a1-4a32-870e-4af50b14d10f" containerName="payload-processing"
Oct 30 21:38:06.736928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.736909    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.740511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.740486    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-2036742396f6\""
Oct 30 21:38:06.740511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.740504    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-e421999e5d93\""
Oct 30 21:38:06.741384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.741352    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-2f614df36ac6\""
Oct 30 21:38:06.741498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.741384    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-plugins-e2e-extmod-b-36a099\""
Oct 30 21:38:06.741498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.741386    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"]
Oct 30 21:38:06.741498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.741421    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-e2e--3e617b48-dockercfg-d4sw2\""
Oct 30 21:38:06.741498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.741451    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"routing-overlay\""
Oct 30 21:38:06.741498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.741457    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"openshift-service-ca.crt\""
Oct 30 21:38:06.741498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.741447    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"kube-root-ca.crt\""
Oct 30 21:38:06.892618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.892580    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.892825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.892764    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.892870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.892826    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.892991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.892971    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxm4x\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-kube-api-access-mxm4x\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.994501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.994403    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxm4x\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-kube-api-access-mxm4x\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.994699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.994652    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.994877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.994854    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.995000 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.994884    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.995590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.995553    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.995736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.995601    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:06.997722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:06.997661    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:07.003202 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:07.003162    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxm4x\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-kube-api-access-mxm4x\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:07.067388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:07.067357    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:07.386946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:07.386914    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"]
Oct 30 21:38:07.387280 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:38:07.387261    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217bcc71_6b02_48b6_b2e2_1a3ac5ee2fc4.slice/crio-c5da281d487223740dcb4303d2db3ebad8490532f891f83bf0b05f6c12756e3b WatchSource:0}: Error finding container c5da281d487223740dcb4303d2db3ebad8490532f891f83bf0b05f6c12756e3b: Status 404 returned error can't find the container with id c5da281d487223740dcb4303d2db3ebad8490532f891f83bf0b05f6c12756e3b
Oct 30 21:38:08.121731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:08.121691    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg" event={"ID":"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4","Type":"ContainerStarted","Data":"3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9"}
Oct 30 21:38:08.122397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:08.122361    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg" event={"ID":"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4","Type":"ContainerStarted","Data":"c5da281d487223740dcb4303d2db3ebad8490532f891f83bf0b05f6c12756e3b"}
Oct 30 21:38:08.122797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:08.122641    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:08.143429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:08.143377    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg" podStartSLOduration=2.143365365 podStartE2EDuration="2.143365365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:08.141510431 +0000 UTC m=+2196.491237827" watchObservedRunningTime="2026-10-30 21:38:08.143365365 +0000 UTC m=+2196.493092785"
Oct 30 21:38:19.149764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:19.149633    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:38:25.087298 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:38:25.087108    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:38:28.201245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:38:28.201208    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:38:55.020025 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:38:55.019987    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:39:09.270652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:09.270611    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"]
Oct 30 21:39:12.090488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:12.090455    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"]
Oct 30 21:39:14.284309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.284235    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg" podUID="217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" containerName="payload-processing" containerID="cri-o://3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9" gracePeriod=25
Oct 30 21:39:14.624541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.624508    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:39:14.711053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.711020    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-plugins-config-volume\") pod \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") "
Oct 30 21:39:14.711231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.711164    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-provider-credentials\") pod \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") "
Oct 30 21:39:14.711282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.711246    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-kube-api-access-mxm4x\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-kube-api-access-mxm4x\") pod \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") "
Oct 30 21:39:14.711361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.711338    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-plugins-config-volume" pod "217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" (UID: "217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:14.711471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.711454    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-routing-overlay-volume\") pod \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\" (UID: \"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4\") "
Oct 30 21:39:14.712075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.711838    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-routing-overlay-volume" pod "217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" (UID: "217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:14.712221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.712151    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.712221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.712164    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.713585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.713562    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-kube-api-access-mxm4x" pod "217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" (UID: "217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4"). InnerVolumeSpecName "kube-api-access-mxm4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:14.713722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.713592    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-provider-credentials" pod "217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" (UID: "217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:14.848314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.847586    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.848314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:14.847635    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxm4x\" (UniqueName: \"kubernetes.io/projected/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4-kube-api-access-mxm4x\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:15.236416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.236383    4633 generic.go:356] "Generic (PLEG): container finished" podID="217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" containerID="3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9" exitCode=0
Oct 30 21:39:15.236619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.236425    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg" event={"ID":"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4","Type":"ContainerDied","Data":"3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9"}
Oct 30 21:39:15.236619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.236442    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"
Oct 30 21:39:15.236619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.236469    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg" event={"ID":"217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4","Type":"ContainerDied","Data":"c5da281d487223740dcb4303d2db3ebad8490532f891f83bf0b05f6c12756e3b"}
Oct 30 21:39:15.236619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.236489    4633 scope.go:122] "RemoveContainer" containerID="3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9"
Oct 30 21:39:15.248300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.248282    4633 scope.go:122] "RemoveContainer" containerID="3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9"
Oct 30 21:39:15.248568 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:39:15.248546    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9\": container with ID starting with 3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9 not found: ID does not exist" containerID="3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9"
Oct 30 21:39:15.248642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.248579    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9"} err="failed to get container status \"3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9\": rpc error: code = NotFound desc = could not find container \"3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9\": container with ID starting with 3efc202685d2715651ac04ba5554000f225c85eb16ce026029b12f5048de16e9 not found: ID does not exist"
Oct 30 21:39:15.266791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.266764    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"]
Oct 30 21:39:15.279550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.279523    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-698d4rxjg"]
Oct 30 21:39:15.873770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:15.873733    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" path="/var/lib/kubelet/pods/217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4/volumes"
Oct 30 21:39:16.294830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.294794    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"]
Oct 30 21:39:16.310697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.296354    4633 cpu_manager.go:414] "containerMap: removing container" podUID="217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" containerName="payload-processing"
Oct 30 21:39:16.310967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.310938    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" containerName="payload-processing"
Oct 30 21:39:16.311365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.311347    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="217bcc71-6b02-48b6-b2e2-1a3ac5ee2fc4" containerName="payload-processing"
Oct 30 21:39:16.352935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.350047    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.358263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.357421    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"]
Oct 30 21:39:16.376691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.364278    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-2f614df36ac6\""
Oct 30 21:39:16.376691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.364474    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-e2e--3e617b48-dockercfg-d4sw2\""
Oct 30 21:39:16.376691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.366244    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"kube-root-ca.crt\""
Oct 30 21:39:16.376691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.366312    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-plugins-e2e-extmod-b-36a099\""
Oct 30 21:39:16.376691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.366410    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"routing-overlay\""
Oct 30 21:39:16.376691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.366465    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"openshift-service-ca.crt\""
Oct 30 21:39:16.611972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.603428    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.611972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.603602    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.612845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.603662    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp66b\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-kube-api-access-wp66b\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.612845 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.612399    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.713996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.713965    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.714165 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.714130    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.714224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.714211    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.714263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.714245    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp66b\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-kube-api-access-wp66b\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.714650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.714629    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.714973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.714950    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.716959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.716938    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.723355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.723328    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp66b\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-kube-api-access-wp66b\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:16.731330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:16.731309    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:17.106055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.104526    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"]
Oct 30 21:39:17.187634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.187572    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr" podUID="7c7dc822-4903-4a99-89cc-799a73549412" containerName="payload-processing" containerID="cri-o://48e6406cd4e0ee2bcf10cc126b5b7a8e23802f5f52647594e788425dec5d46d7" gracePeriod=25
Oct 30 21:39:17.348268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.348224    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz" event={"ID":"12ced86c-b99c-4009-aa23-521988966383","Type":"ContainerStarted","Data":"7155ed7b60ae16877a94954797181554142a0dc398c0ea047dd8c66f10e0cece"}
Oct 30 21:39:17.348483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.348298    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz" event={"ID":"12ced86c-b99c-4009-aa23-521988966383","Type":"ContainerStarted","Data":"5bfb74c5e154a9db71db3f0b3c396e7c68d637d4d8b371788070f85cb92c4b7f"}
Oct 30 21:39:17.348483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.348327    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:17.352241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.351021    4633 generic.go:356] "Generic (PLEG): container finished" podID="7c7dc822-4903-4a99-89cc-799a73549412" containerID="48e6406cd4e0ee2bcf10cc126b5b7a8e23802f5f52647594e788425dec5d46d7" exitCode=0
Oct 30 21:39:17.352241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.351152    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr" event={"ID":"7c7dc822-4903-4a99-89cc-799a73549412","Type":"ContainerDied","Data":"48e6406cd4e0ee2bcf10cc126b5b7a8e23802f5f52647594e788425dec5d46d7"}
Oct 30 21:39:17.373855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.373799    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz" podStartSLOduration=1.373782055 podStartE2EDuration="1.373782055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:17.371484131 +0000 UTC m=+2265.721211528" watchObservedRunningTime="2026-10-30 21:39:17.373782055 +0000 UTC m=+2265.723509481"
Oct 30 21:39:17.476558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.476528    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:39:17.658759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.647232    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-kube-api-access-f7kls\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-kube-api-access-f7kls\") pod \"7c7dc822-4903-4a99-89cc-799a73549412\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") "
Oct 30 21:39:17.658759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.647497    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-plugins-config-volume\") pod \"7c7dc822-4903-4a99-89cc-799a73549412\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") "
Oct 30 21:39:17.658759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.647589    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-provider-credentials\") pod \"7c7dc822-4903-4a99-89cc-799a73549412\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") "
Oct 30 21:39:17.658759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.647716    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-routing-overlay-volume\") pod \"7c7dc822-4903-4a99-89cc-799a73549412\" (UID: \"7c7dc822-4903-4a99-89cc-799a73549412\") "
Oct 30 21:39:17.674702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.667979    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-routing-overlay-volume" pod "7c7dc822-4903-4a99-89cc-799a73549412" (UID: "7c7dc822-4903-4a99-89cc-799a73549412"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:17.674702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.668324    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-plugins-config-volume" pod "7c7dc822-4903-4a99-89cc-799a73549412" (UID: "7c7dc822-4903-4a99-89cc-799a73549412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:17.685954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.675628    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:17.708267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.708229    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c7dc822-4903-4a99-89cc-799a73549412-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:17.863735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.862424    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-kube-api-access-f7kls" pod "7c7dc822-4903-4a99-89cc-799a73549412" (UID: "7c7dc822-4903-4a99-89cc-799a73549412"). InnerVolumeSpecName "kube-api-access-f7kls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:17.863735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.863006    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-provider-credentials" pod "7c7dc822-4903-4a99-89cc-799a73549412" (UID: "7c7dc822-4903-4a99-89cc-799a73549412"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:17.945199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.945100    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:17.945199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:17.945141    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7kls\" (UniqueName: \"kubernetes.io/projected/7c7dc822-4903-4a99-89cc-799a73549412-kube-api-access-f7kls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:18.361101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:18.361019    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"
Oct 30 21:39:18.361101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:18.361050    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr" event={"ID":"7c7dc822-4903-4a99-89cc-799a73549412","Type":"ContainerDied","Data":"605cef4d3ffe53d7171ad1bc99d6227071fec3fa037fbe80af81bde61c75b41d"}
Oct 30 21:39:18.361637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:18.361115    4633 scope.go:122] "RemoveContainer" containerID="48e6406cd4e0ee2bcf10cc126b5b7a8e23802f5f52647594e788425dec5d46d7"
Oct 30 21:39:18.404241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:18.404212    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"]
Oct 30 21:39:18.431207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:18.431179    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-5c579lrrr"]
Oct 30 21:39:19.793380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:19.793346    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7dc822-4903-4a99-89cc-799a73549412" path="/var/lib/kubelet/pods/7c7dc822-4903-4a99-89cc-799a73549412/volumes"
Oct 30 21:39:25.093149 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:39:25.093090    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:39:26.028278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.028244    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"]
Oct 30 21:39:26.029069 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.029052    4633 cpu_manager.go:414] "containerMap: removing container" podUID="7c7dc822-4903-4a99-89cc-799a73549412" containerName="payload-processing"
Oct 30 21:39:26.029144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.029070    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7dc822-4903-4a99-89cc-799a73549412" containerName="payload-processing"
Oct 30 21:39:26.029243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.029233    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7dc822-4903-4a99-89cc-799a73549412" containerName="payload-processing"
Oct 30 21:39:26.033303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.033283    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.041174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.041011    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:39:26.041174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.041013    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fa4045c246e1\""
Oct 30 21:39:26.041174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.041022    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"]
Oct 30 21:39:26.041174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.041084    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:39:26.041174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.041089    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-76921397fa76\""
Oct 30 21:39:26.041174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.041110    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-jgk7c\""
Oct 30 21:39:26.041174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.041089    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:39:26.041174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.041155    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:39:26.136907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.136872    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-plugins-config-volume\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.137332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.136997    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-routing-overlay-volume\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.137332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.137057    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwr94\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-kube-api-access-cwr94\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.137420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.137327    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-provider-credentials\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.238520 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.238485    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-routing-overlay-volume\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.238775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.238618    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwr94\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-kube-api-access-cwr94\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.238883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.238864    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-provider-credentials\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.238980 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.238963    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-plugins-config-volume\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.239330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.239303    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-routing-overlay-volume\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.239767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.239744    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-plugins-config-volume\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.241630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.241607    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-provider-credentials\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.248428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.248407    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwr94\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-kube-api-access-cwr94\") pod \"payload-processing-external-model-b86448fc9-vlqkj\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") " pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.366753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.361974    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:26.677511 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:39:26.677481    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb109217_fd55_47ee_9d82_2433e53905a5.slice/crio-fc0ffea223792aebebeaff886afcfc402fb5ae8dc3f547dc56d209de92cc0e3e WatchSource:0}: Error finding container fc0ffea223792aebebeaff886afcfc402fb5ae8dc3f547dc56d209de92cc0e3e: Status 404 returned error can't find the container with id fc0ffea223792aebebeaff886afcfc402fb5ae8dc3f547dc56d209de92cc0e3e
Oct 30 21:39:26.677799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:26.677776    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"]
Oct 30 21:39:27.497785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:27.497750    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj" event={"ID":"cb109217-fd55-47ee-9d82-2433e53905a5","Type":"ContainerStarted","Data":"3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097"}
Oct 30 21:39:27.498325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:27.497841    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj" event={"ID":"cb109217-fd55-47ee-9d82-2433e53905a5","Type":"ContainerStarted","Data":"fc0ffea223792aebebeaff886afcfc402fb5ae8dc3f547dc56d209de92cc0e3e"}
Oct 30 21:39:27.498325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:27.498033    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:27.516453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:27.516403    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj" podStartSLOduration=1.516389358 podStartE2EDuration="1.516389358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:27.514829146 +0000 UTC m=+2275.864556553" watchObservedRunningTime="2026-10-30 21:39:27.516389358 +0000 UTC m=+2275.866116754"
Oct 30 21:39:27.982693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:27.981955    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:39:28.365535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:28.365460    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:39:38.511080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:38.511041    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:39:47.300828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.300729    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"]
Oct 30 21:39:47.324696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.319380    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.325296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.325265    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"]
Oct 30 21:39:47.438059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.438016    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.438507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.438460    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdcd\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-kube-api-access-bwdcd\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.438728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.438593    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.438728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.438644    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.438885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.438770    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.438885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.438817    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf3712-d04c-4816-9a32-3de435a3f6f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.438968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.438908    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.438968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.438954    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.439043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.439010    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.540916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.540880    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.540938    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf3712-d04c-4816-9a32-3de435a3f6f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.540974    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.540996    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541051    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541164    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541277    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdcd\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-kube-api-access-bwdcd\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541396    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541438    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541482    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541528    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541582    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.541800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.541753    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.542171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.542084    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf3712-d04c-4816-9a32-3de435a3f6f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.543971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.543946    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.544171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.544154    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.549292 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.549270    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.549465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.549443    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdcd\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-kube-api-access-bwdcd\") pod \"maas-default-gateway-openshift-default-75bf867b6c-4lkpp\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.645991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.645900    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:47.942479 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:39:47.942448    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cf3712_d04c_4816_9a32_3de435a3f6f7.slice/crio-a53be1cd566c3f8b11da899f9d963ae29eae95bc56bf68d9f46ad2b31c7c06af WatchSource:0}: Error finding container a53be1cd566c3f8b11da899f9d963ae29eae95bc56bf68d9f46ad2b31c7c06af: Status 404 returned error can't find the container with id a53be1cd566c3f8b11da899f9d963ae29eae95bc56bf68d9f46ad2b31c7c06af
Oct 30 21:39:47.943604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.943576    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"]
Oct 30 21:39:47.945794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.945751    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:39:47.945906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.945857    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:39:47.945906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:47.945895    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:39:48.052641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:48.052605    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" event={"ID":"90cf3712-d04c-4816-9a32-3de435a3f6f7","Type":"ContainerStarted","Data":"a53be1cd566c3f8b11da899f9d963ae29eae95bc56bf68d9f46ad2b31c7c06af"}
Oct 30 21:39:49.062837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:49.062802    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" event={"ID":"90cf3712-d04c-4816-9a32-3de435a3f6f7","Type":"ContainerStarted","Data":"11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9"}
Oct 30 21:39:49.095658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:49.095598    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podStartSLOduration=2.095580126 podStartE2EDuration="2.095580126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:49.08980236 +0000 UTC m=+2297.439529759" watchObservedRunningTime="2026-10-30 21:39:49.095580126 +0000 UTC m=+2297.445307524"
Oct 30 21:39:49.646578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:49.646540    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:49.649577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:49.649544    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4lkpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:39:49.649901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:49.649867    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 21:39:50.647016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:50.646979    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4lkpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:39:50.647684 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:50.647630    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 21:39:51.647216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:51.647173    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4lkpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:39:51.647775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:51.647236    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 21:39:52.646519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:52.646484    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4lkpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:39:52.646777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:52.646542    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 21:39:53.646363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:53.646321    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4lkpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:39:53.646818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:53.646378    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 21:39:54.646778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:54.646723    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4lkpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:39:54.647204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:54.646805    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 21:39:55.020911 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:39:55.020866    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:39:55.646824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:55.646781    4633 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-4lkpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:39:55.647245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:55.646851    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 21:39:57.498408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:57.498375    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:57.500043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:57.500019    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:57.500830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:57.500813    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:39:59.167628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:39:59.167585    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"]
Oct 30 21:40:04.207048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.206995    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz" podUID="12ced86c-b99c-4009-aa23-521988966383" containerName="payload-processing" containerID="cri-o://7155ed7b60ae16877a94954797181554142a0dc398c0ea047dd8c66f10e0cece" gracePeriod=25
Oct 30 21:40:04.424638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.424605    4633 generic.go:356] "Generic (PLEG): container finished" podID="12ced86c-b99c-4009-aa23-521988966383" containerID="7155ed7b60ae16877a94954797181554142a0dc398c0ea047dd8c66f10e0cece" exitCode=0
Oct 30 21:40:04.424868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.424700    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz" event={"ID":"12ced86c-b99c-4009-aa23-521988966383","Type":"ContainerDied","Data":"7155ed7b60ae16877a94954797181554142a0dc398c0ea047dd8c66f10e0cece"}
Oct 30 21:40:04.587542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.587518    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:40:04.718714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.712741    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-routing-overlay-volume\") pod \"12ced86c-b99c-4009-aa23-521988966383\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") "
Oct 30 21:40:04.718714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.713135    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-routing-overlay-volume" pod "12ced86c-b99c-4009-aa23-521988966383" (UID: "12ced86c-b99c-4009-aa23-521988966383"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:04.718714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.714263    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-plugins-config-volume\") pod \"12ced86c-b99c-4009-aa23-521988966383\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") "
Oct 30 21:40:04.718714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.714998    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-plugins-config-volume" pod "12ced86c-b99c-4009-aa23-521988966383" (UID: "12ced86c-b99c-4009-aa23-521988966383"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:04.718714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.715073    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-provider-credentials\") pod \"12ced86c-b99c-4009-aa23-521988966383\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") "
Oct 30 21:40:04.718714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.715241    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-kube-api-access-wp66b\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-kube-api-access-wp66b\") pod \"12ced86c-b99c-4009-aa23-521988966383\" (UID: \"12ced86c-b99c-4009-aa23-521988966383\") "
Oct 30 21:40:04.718714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.716324    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:04.718714 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.716349    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12ced86c-b99c-4009-aa23-521988966383-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:04.795574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.792401    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-provider-credentials" pod "12ced86c-b99c-4009-aa23-521988966383" (UID: "12ced86c-b99c-4009-aa23-521988966383"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:04.795574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.792449    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-kube-api-access-wp66b" pod "12ced86c-b99c-4009-aa23-521988966383" (UID: "12ced86c-b99c-4009-aa23-521988966383"). InnerVolumeSpecName "kube-api-access-wp66b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:04.817484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.817410    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:04.817484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:04.817439    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp66b\" (UniqueName: \"kubernetes.io/projected/12ced86c-b99c-4009-aa23-521988966383-kube-api-access-wp66b\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:05.435296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:05.435265    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"
Oct 30 21:40:05.435803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:05.435298    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz" event={"ID":"12ced86c-b99c-4009-aa23-521988966383","Type":"ContainerDied","Data":"5bfb74c5e154a9db71db3f0b3c396e7c68d637d4d8b371788070f85cb92c4b7f"}
Oct 30 21:40:05.435803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:05.435359    4633 scope.go:122] "RemoveContainer" containerID="7155ed7b60ae16877a94954797181554142a0dc398c0ea047dd8c66f10e0cece"
Oct 30 21:40:05.509985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:05.509952    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"]
Oct 30 21:40:05.520653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:05.520625    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccn8rgz"]
Oct 30 21:40:05.797775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:05.797099    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ced86c-b99c-4009-aa23-521988966383" path="/var/lib/kubelet/pods/12ced86c-b99c-4009-aa23-521988966383/volumes"
Oct 30 21:40:25.074703 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:40:25.073868    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:40:27.356680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:27.356628    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"]
Oct 30 21:40:28.272337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:28.272302    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:40:32.380051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:32.379965    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj" podUID="cb109217-fd55-47ee-9d82-2433e53905a5" containerName="payload-processing" containerID="cri-o://3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097" gracePeriod=25
Oct 30 21:40:33.878013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:33.877983    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:40:34.003794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.003756    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-routing-overlay-volume\") pod \"cb109217-fd55-47ee-9d82-2433e53905a5\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") "
Oct 30 21:40:34.003977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.003852    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-kube-api-access-cwr94\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-kube-api-access-cwr94\") pod \"cb109217-fd55-47ee-9d82-2433e53905a5\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") "
Oct 30 21:40:34.004039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.004011    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-plugins-config-volume\") pod \"cb109217-fd55-47ee-9d82-2433e53905a5\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") "
Oct 30 21:40:34.004176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.004154    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-routing-overlay-volume" pod "cb109217-fd55-47ee-9d82-2433e53905a5" (UID: "cb109217-fd55-47ee-9d82-2433e53905a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:34.004325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.004307    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-provider-credentials\") pod \"cb109217-fd55-47ee-9d82-2433e53905a5\" (UID: \"cb109217-fd55-47ee-9d82-2433e53905a5\") "
Oct 30 21:40:34.004419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.004394    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-plugins-config-volume" pod "cb109217-fd55-47ee-9d82-2433e53905a5" (UID: "cb109217-fd55-47ee-9d82-2433e53905a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:34.005013 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.004991    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:34.005133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.005016    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb109217-fd55-47ee-9d82-2433e53905a5-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:34.006179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.006149    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-kube-api-access-cwr94" pod "cb109217-fd55-47ee-9d82-2433e53905a5" (UID: "cb109217-fd55-47ee-9d82-2433e53905a5"). InnerVolumeSpecName "kube-api-access-cwr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:34.006467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.006446    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-provider-credentials" pod "cb109217-fd55-47ee-9d82-2433e53905a5" (UID: "cb109217-fd55-47ee-9d82-2433e53905a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:34.106640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.106599    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:34.106640 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.106635    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwr94\" (UniqueName: \"kubernetes.io/projected/cb109217-fd55-47ee-9d82-2433e53905a5-kube-api-access-cwr94\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:34.124113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.124078    4633 generic.go:356] "Generic (PLEG): container finished" podID="cb109217-fd55-47ee-9d82-2433e53905a5" containerID="3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097" exitCode=0
Oct 30 21:40:34.124288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.124129    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"
Oct 30 21:40:34.124288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.124249    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj" event={"ID":"cb109217-fd55-47ee-9d82-2433e53905a5","Type":"ContainerDied","Data":"3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097"}
Oct 30 21:40:34.124288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.124290    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj" event={"ID":"cb109217-fd55-47ee-9d82-2433e53905a5","Type":"ContainerDied","Data":"fc0ffea223792aebebeaff886afcfc402fb5ae8dc3f547dc56d209de92cc0e3e"}
Oct 30 21:40:34.124410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.124317    4633 scope.go:122] "RemoveContainer" containerID="3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097"
Oct 30 21:40:34.138497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.138479    4633 scope.go:122] "RemoveContainer" containerID="3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097"
Oct 30 21:40:34.138840 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:40:34.138820    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097\": container with ID starting with 3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097 not found: ID does not exist" containerID="3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097"
Oct 30 21:40:34.138910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.138849    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097"} err="failed to get container status \"3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097\": rpc error: code = NotFound desc = could not find container \"3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097\": container with ID starting with 3695fedeaf1513b86923d98e164fb386237f50c26da33a221d66b32b4f703097 not found: ID does not exist"
Oct 30 21:40:34.150900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.150868    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"]
Oct 30 21:40:34.155889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:34.155863    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-b86448fc9-vlqkj"]
Oct 30 21:40:35.044413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:35.044383    4633 scope.go:122] "RemoveContainer" containerID="b3739d4df6ea7065974c26a2bfbe9eff78c462aa91328a030f08431363222ca2"
Oct 30 21:40:35.793545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:40:35.793509    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb109217-fd55-47ee-9d82-2433e53905a5" path="/var/lib/kubelet/pods/cb109217-fd55-47ee-9d82-2433e53905a5/volumes"
Oct 30 21:40:55.017719 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:40:55.017656    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:41:03.602321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:03.602051    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"]
Oct 30 21:41:03.603012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:03.602427    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" containerID="cri-o://a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f" gracePeriod=30
Oct 30 21:41:09.175042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.175007    4633 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:41:09.175460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.175070    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 21:41:09.299463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.299436    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:41:09.415067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415036    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-certs\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415149    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-credential-socket\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415171    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-kube-api-access-rsrpf\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-kube-api-access-rsrpf\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415189    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07f6b01-10e6-4f92-a40f-87f2d6154725-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f6b01-10e6-4f92-a40f-87f2d6154725-istiod-ca-cert\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415214    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-envoy\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415281    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-token\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-token\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415335    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-certs" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:09.415431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415390    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-socket\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415485    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-data\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415514    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-credential-socket" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:09.415605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415534    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-podinfo\") pod \"a07f6b01-10e6-4f92-a40f-87f2d6154725\" (UID: \"a07f6b01-10e6-4f92-a40f-87f2d6154725\") "
Oct 30 21:41:09.415811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415620    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07f6b01-10e6-4f92-a40f-87f2d6154725-istiod-ca-cert" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:09.415811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.415740    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-data" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:09.416133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.416087    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-socket" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:09.416359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.416342    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.416435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.416361    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f6b01-10e6-4f92-a40f-87f2d6154725-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.416435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.416371    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.416435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.416379    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.416435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.416388    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.417871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.417844    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-token" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:09.417984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.417901    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-envoy" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:09.418248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.418225    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-podinfo" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:41:09.418351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.418252    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-kube-api-access-rsrpf" pod "a07f6b01-10e6-4f92-a40f-87f2d6154725" (UID: "a07f6b01-10e6-4f92-a40f-87f2d6154725"). InnerVolumeSpecName "kube-api-access-rsrpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:09.526088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.523152    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.526088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.523195    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrpf\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-kube-api-access-rsrpf\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.526088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.523211    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.526088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.523226    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f6b01-10e6-4f92-a40f-87f2d6154725-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:09.557338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.557304    4633 generic.go:356] "Generic (PLEG): container finished" podID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerID="a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f" exitCode=0
Oct 30 21:41:09.557550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.557353    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"
Oct 30 21:41:09.557550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.557396    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" event={"ID":"a07f6b01-10e6-4f92-a40f-87f2d6154725","Type":"ContainerDied","Data":"a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f"}
Oct 30 21:41:09.557550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.557462    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd" event={"ID":"a07f6b01-10e6-4f92-a40f-87f2d6154725","Type":"ContainerDied","Data":"487378a02c27ce38e969ac21e9b4ce6d8ca539d962424a079103a4adf0b03e6b"}
Oct 30 21:41:09.557550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.557493    4633 scope.go:122] "RemoveContainer" containerID="a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f"
Oct 30 21:41:09.588826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.588785    4633 scope.go:122] "RemoveContainer" containerID="a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f"
Oct 30 21:41:09.589223 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:09.589196    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f\": container with ID starting with a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f not found: ID does not exist" containerID="a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f"
Oct 30 21:41:09.589302 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.589233    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f"} err="failed to get container status \"a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f\": rpc error: code = NotFound desc = could not find container \"a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f\": container with ID starting with a8bd675d600cd426a9d1396097f595eb6cc6c3f45f2cc84d4500d5d28d12ab2f not found: ID does not exist"
Oct 30 21:41:09.592103 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.592083    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"]
Oct 30 21:41:09.601145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.601103    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-c9qhd"]
Oct 30 21:41:09.793245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:09.793170    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" path="/var/lib/kubelet/pods/a07f6b01-10e6-4f92-a40f-87f2d6154725/volumes"
Oct 30 21:41:11.430020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:11.429990    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"]
Oct 30 21:41:11.577419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:11.577375    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy" containerID="cri-o://4857af0cab47837ec01a777def7f51fe7a7197eb4d30bb85a7a0edb65c97e27b" gracePeriod=30
Oct 30 21:41:15.932762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:15.932659    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:41:16.722937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:16.722903    4633 generic.go:356] "Generic (PLEG): container finished" podID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerID="4857af0cab47837ec01a777def7f51fe7a7197eb4d30bb85a7a0edb65c97e27b" exitCode=0
Oct 30 21:41:16.723155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:16.722968    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" event={"ID":"5778ea70-23d5-467d-bd3d-e034ef9327dc","Type":"ContainerDied","Data":"4857af0cab47837ec01a777def7f51fe7a7197eb4d30bb85a7a0edb65c97e27b"}
Oct 30 21:41:16.723155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:16.723046    4633 scope.go:122] "RemoveContainer" containerID="b992455bdf9e7e4ab1d167e7d8d63c57cf084d05bbab877b475ead547ac2f1e3"
Oct 30 21:41:17.527696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.525211    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:41:17.754635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.736645    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-kube-api-access-5g82t\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-kube-api-access-5g82t\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.754635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.736771    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-credential-socket\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.754635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.736812    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-certs\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.754635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.736932    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-podinfo\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.754635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.737055    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-data\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.755883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.755471    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-data" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:17.756690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.756401    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-credential-socket" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:17.758766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.756852    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-certs" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:17.822837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.822801    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5778ea70-23d5-467d-bd3d-e034ef9327dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778ea70-23d5-467d-bd3d-e034ef9327dc-istiod-ca-cert\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.823001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.822915    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-token\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-token\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.823001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.822991    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-envoy\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.823126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.823041    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-socket\") pod \"5778ea70-23d5-467d-bd3d-e034ef9327dc\" (UID: \"5778ea70-23d5-467d-bd3d-e034ef9327dc\") "
Oct 30 21:41:17.824443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.824415    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.824560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.824449    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.824560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.824473    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.824769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.824745    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-socket" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:17.844152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.844048    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5778ea70-23d5-467d-bd3d-e034ef9327dc-istiod-ca-cert" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:17.845661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.845356    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-kube-api-access-5g82t" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "kube-api-access-5g82t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:17.865022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.859064    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-envoy" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:17.885928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.878965    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-token" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:17.885928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.872867    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-podinfo" pod "5778ea70-23d5-467d-bd3d-e034ef9327dc" (UID: "5778ea70-23d5-467d-bd3d-e034ef9327dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:41:17.917695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.915689    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt" event={"ID":"5778ea70-23d5-467d-bd3d-e034ef9327dc","Type":"ContainerDied","Data":"a1339a357d2a875d745e43d65532e3baa356e161b8357c592b2d9eac8276f40d"}
Oct 30 21:41:17.917695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.915773    4633 scope.go:122] "RemoveContainer" containerID="4857af0cab47837ec01a777def7f51fe7a7197eb4d30bb85a7a0edb65c97e27b"
Oct 30 21:41:17.917695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.915935    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"
Oct 30 21:41:17.928179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.927146    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.928179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.927175    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5778ea70-23d5-467d-bd3d-e034ef9327dc-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.928179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.927191    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g82t\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-kube-api-access-5g82t\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.928179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.927208    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.928179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.927222    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5778ea70-23d5-467d-bd3d-e034ef9327dc-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.928179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.927235    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5778ea70-23d5-467d-bd3d-e034ef9327dc-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:17.975700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.972942    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"]
Oct 30 21:41:17.983701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:17.982461    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gpwtt"]
Oct 30 21:41:19.816559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:19.816518    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" path="/var/lib/kubelet/pods/5778ea70-23d5-467d-bd3d-e034ef9327dc/volumes"
Oct 30 21:41:25.015889 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:25.015852    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:41:25.538990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.538960    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"]
Oct 30 21:41:25.635155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635117    4633 cpu_manager.go:414] "containerMap: removing container" podUID="12ced86c-b99c-4009-aa23-521988966383" containerName="payload-processing"
Oct 30 21:41:25.635155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635156    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ced86c-b99c-4009-aa23-521988966383" containerName="payload-processing"
Oct 30 21:41:25.635362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635175    4633 cpu_manager.go:414] "containerMap: removing container" podUID="cb109217-fd55-47ee-9d82-2433e53905a5" containerName="payload-processing"
Oct 30 21:41:25.635362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635183    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb109217-fd55-47ee-9d82-2433e53905a5" containerName="payload-processing"
Oct 30 21:41:25.635362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635241    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy"
Oct 30 21:41:25.635362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635249    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy"
Oct 30 21:41:25.635362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635289    4633 cpu_manager.go:414] "containerMap: removing container" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy"
Oct 30 21:41:25.635362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635297    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy"
Oct 30 21:41:25.635362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635315    4633 cpu_manager.go:414] "containerMap: removing container" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy"
Oct 30 21:41:25.635362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635322    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy"
Oct 30 21:41:25.635610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635543    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy"
Oct 30 21:41:25.635610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635562    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5778ea70-23d5-467d-bd3d-e034ef9327dc" containerName="istio-proxy"
Oct 30 21:41:25.635610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635583    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ced86c-b99c-4009-aa23-521988966383" containerName="payload-processing"
Oct 30 21:41:25.635610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635593    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb109217-fd55-47ee-9d82-2433e53905a5" containerName="payload-processing"
Oct 30 21:41:25.635610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.635608    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07f6b01-10e6-4f92-a40f-87f2d6154725" containerName="istio-proxy"
Oct 30 21:41:25.643102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.643075    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"]
Oct 30 21:41:25.643285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.643198    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.647997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.647150    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"agc-cred-64ea8ebbfd9e\""
Oct 30 21:41:25.647997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.647418    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-e2e--57c066ea-dockercfg-nwq84\""
Oct 30 21:41:25.648251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.648232    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"openshift-service-ca.crt\""
Oct 30 21:41:25.648359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.648339    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"kube-root-ca.crt\""
Oct 30 21:41:25.648543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.648446    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"routing-overlay\""
Oct 30 21:41:25.648649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.648554    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e596df\""
Oct 30 21:41:25.726529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.726494    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4c5b\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-kube-api-access-q4c5b\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.726695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.726644    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.726880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.726865    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.726973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.726957    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.828413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.828315    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.828413 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.828407    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4c5b\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-kube-api-access-q4c5b\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.828711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.828486    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.828711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.828634    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.829232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.829208    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.829310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.829290    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.831243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.831217    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.839163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.839144    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4c5b\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-kube-api-access-q4c5b\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:25.964472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:25.964440    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:26.257761 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:41:26.257726    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe7aaf70_3b8b_45b6_b427_2305419c974c.slice/crio-d6cf9ba2844a1580796e1a5ae1b915a62812022578e4a6308a8f9739652fc52c WatchSource:0}: Error finding container d6cf9ba2844a1580796e1a5ae1b915a62812022578e4a6308a8f9739652fc52c: Status 404 returned error can't find the container with id d6cf9ba2844a1580796e1a5ae1b915a62812022578e4a6308a8f9739652fc52c
Oct 30 21:41:26.258438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:26.258320    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"]
Oct 30 21:41:26.423727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:26.422826    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"]
Oct 30 21:41:27.128697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:27.128313    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5" event={"ID":"fe7aaf70-3b8b-45b6-b427-2305419c974c","Type":"ContainerStarted","Data":"ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092"}
Oct 30 21:41:27.128697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:27.128364    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5" event={"ID":"fe7aaf70-3b8b-45b6-b427-2305419c974c","Type":"ContainerStarted","Data":"d6cf9ba2844a1580796e1a5ae1b915a62812022578e4a6308a8f9739652fc52c"}
Oct 30 21:41:27.128930 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:27.128850    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:27.152347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:27.152293    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5" podStartSLOduration=2.152277445 podStartE2EDuration="2.152277445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:27.14658144 +0000 UTC m=+2395.496308836" watchObservedRunningTime="2026-10-30 21:41:27.152277445 +0000 UTC m=+2395.502004841"
Oct 30 21:41:32.128998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.128933    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5" podUID="fe7aaf70-3b8b-45b6-b427-2305419c974c" containerName="payload-processing" containerID="cri-o://ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092" gracePeriod=25
Oct 30 21:41:32.450490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.450462    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:32.457475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.457444    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-plugins-config-volume\") pod \"fe7aaf70-3b8b-45b6-b427-2305419c974c\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") "
Oct 30 21:41:32.457654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.457552    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-kube-api-access-q4c5b\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-kube-api-access-q4c5b\") pod \"fe7aaf70-3b8b-45b6-b427-2305419c974c\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") "
Oct 30 21:41:32.457654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.457579    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-routing-overlay-volume\") pod \"fe7aaf70-3b8b-45b6-b427-2305419c974c\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") "
Oct 30 21:41:32.457654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.457615    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-provider-credentials\") pod \"fe7aaf70-3b8b-45b6-b427-2305419c974c\" (UID: \"fe7aaf70-3b8b-45b6-b427-2305419c974c\") "
Oct 30 21:41:32.457995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.457968    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-plugins-config-volume" pod "fe7aaf70-3b8b-45b6-b427-2305419c974c" (UID: "fe7aaf70-3b8b-45b6-b427-2305419c974c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:32.458146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.458018    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-routing-overlay-volume" pod "fe7aaf70-3b8b-45b6-b427-2305419c974c" (UID: "fe7aaf70-3b8b-45b6-b427-2305419c974c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:32.458421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.458402    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:32.458497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.458428    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fe7aaf70-3b8b-45b6-b427-2305419c974c-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:32.460206 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.460180    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-provider-credentials" pod "fe7aaf70-3b8b-45b6-b427-2305419c974c" (UID: "fe7aaf70-3b8b-45b6-b427-2305419c974c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:32.460296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.460220    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-kube-api-access-q4c5b" pod "fe7aaf70-3b8b-45b6-b427-2305419c974c" (UID: "fe7aaf70-3b8b-45b6-b427-2305419c974c"). InnerVolumeSpecName "kube-api-access-q4c5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:32.586873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.579916    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4c5b\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-kube-api-access-q4c5b\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:32.586873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:32.579962    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fe7aaf70-3b8b-45b6-b427-2305419c974c-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:33.210548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.210476    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:41:33.214008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.213985    4633 generic.go:356] "Generic (PLEG): container finished" podID="fe7aaf70-3b8b-45b6-b427-2305419c974c" containerID="ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092" exitCode=0
Oct 30 21:41:33.214203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.214185    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5" event={"ID":"fe7aaf70-3b8b-45b6-b427-2305419c974c","Type":"ContainerDied","Data":"ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092"}
Oct 30 21:41:33.214325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.214312    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5" event={"ID":"fe7aaf70-3b8b-45b6-b427-2305419c974c","Type":"ContainerDied","Data":"d6cf9ba2844a1580796e1a5ae1b915a62812022578e4a6308a8f9739652fc52c"}
Oct 30 21:41:33.214423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.214413    4633 scope.go:122] "RemoveContainer" containerID="ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092"
Oct 30 21:41:33.214740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.214725    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"
Oct 30 21:41:33.231273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.231254    4633 scope.go:122] "RemoveContainer" containerID="ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092"
Oct 30 21:41:33.231558 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:33.231532    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092\": container with ID starting with ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092 not found: ID does not exist" containerID="ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092"
Oct 30 21:41:33.231629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.231571    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092"} err="failed to get container status \"ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092\": rpc error: code = NotFound desc = could not find container \"ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092\": container with ID starting with ab4d6d450e839ff0ff50ab43823f7f7dc5e2688d12687a94125554813bab3092 not found: ID does not exist"
Oct 30 21:41:33.234771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.234746    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:41:33.237580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.237558    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:41:33.250270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.250236    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"]
Oct 30 21:41:33.255814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.255791    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:41:33.256094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.256077    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7tq7m5"]
Oct 30 21:41:33.818049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:33.818019    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7aaf70-3b8b-45b6-b427-2305419c974c" path="/var/lib/kubelet/pods/fe7aaf70-3b8b-45b6-b427-2305419c974c/volumes"
Oct 30 21:41:43.336123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.336089    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"]
Oct 30 21:41:43.336967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.336951    4633 cpu_manager.go:414] "containerMap: removing container" podUID="fe7aaf70-3b8b-45b6-b427-2305419c974c" containerName="payload-processing"
Oct 30 21:41:43.337027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.336969    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7aaf70-3b8b-45b6-b427-2305419c974c" containerName="payload-processing"
Oct 30 21:41:43.337098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.337089    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7aaf70-3b8b-45b6-b427-2305419c974c" containerName="payload-processing"
Oct 30 21:41:43.342870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.342849    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.350082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.350052    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"openshift-service-ca.crt\""
Oct 30 21:41:43.350257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.350130    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"kube-root-ca.crt\""
Oct 30 21:41:43.350257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.350052    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"routing-overlay\""
Oct 30 21:41:43.350257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.350130    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"agc-cred-ceaae6de1e09\""
Oct 30 21:41:43.350257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.350190    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e596df\""
Oct 30 21:41:43.350257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.350209    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-e2e--57c066ea-dockercfg-gmr2l\""
Oct 30 21:41:43.356775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.356747    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"]
Oct 30 21:41:43.396450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.396415    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.396632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.396496    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29dt\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-kube-api-access-x29dt\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.396632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.396610    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.396861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.396839    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.505694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.498192    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29dt\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-kube-api-access-x29dt\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.505694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.498263    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.505694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.498454    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.505694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.498716    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.535718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.530588    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.564636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.564599    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.569869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.569803    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.619776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.613710    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29dt\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-kube-api-access-x29dt\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:43.663009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:43.662967    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:44.166344 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:41:44.166313    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4de7390_e80b_4c49_84a8_b178ecb8a27d.slice/crio-4f9ab60f40faaad2e54adcbeda7c1ff7c5058c0473c312d50b76520db9f5b18f WatchSource:0}: Error finding container 4f9ab60f40faaad2e54adcbeda7c1ff7c5058c0473c312d50b76520db9f5b18f: Status 404 returned error can't find the container with id 4f9ab60f40faaad2e54adcbeda7c1ff7c5058c0473c312d50b76520db9f5b18f
Oct 30 21:41:44.166978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:44.166956    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"]
Oct 30 21:41:44.372446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:44.371467    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g" event={"ID":"a4de7390-e80b-4c49-84a8-b178ecb8a27d","Type":"ContainerStarted","Data":"ab4753438ab9c9c12d856b642a06e90cc660421573f247653e6fccf487f17ae7"}
Oct 30 21:41:44.372446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:44.371540    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g" event={"ID":"a4de7390-e80b-4c49-84a8-b178ecb8a27d","Type":"ContainerStarted","Data":"4f9ab60f40faaad2e54adcbeda7c1ff7c5058c0473c312d50b76520db9f5b18f"}
Oct 30 21:41:44.372446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:44.371931    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:44.406733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:44.403577    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g" podStartSLOduration=1.403558688 podStartE2EDuration="1.403558688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:44.400972496 +0000 UTC m=+2412.750699893" watchObservedRunningTime="2026-10-30 21:41:44.403558688 +0000 UTC m=+2412.753286085"
Oct 30 21:41:48.875779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.875737    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"]
Oct 30 21:41:48.880738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.880708    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:48.887524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.887497    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-76921397fa76\""
Oct 30 21:41:48.887689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.887605    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:41:48.888540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.888516    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5ac5f2767576\""
Oct 30 21:41:48.888662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.888625    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ca27eaecc19b\""
Oct 30 21:41:48.888662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.888690    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:41:48.889408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.888754    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:41:48.889408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.888791    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fa4045c246e1\""
Oct 30 21:41:48.894398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.894372    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"]
Oct 30 21:41:48.904919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.904881    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:41:48.954743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.954700    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-routing-overlay-volume\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:48.954952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.954834    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-plugins-config-volume\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:48.955059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.954883    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:48.955315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.955300    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttvb\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-kube-api-access-jttvb\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:48.986927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:48.986891    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-jgk7c\""
Oct 30 21:41:49.056517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.056480    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttvb\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-kube-api-access-jttvb\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:49.056774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.056711    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-routing-overlay-volume\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:49.056866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.056821    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-plugins-config-volume\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:49.056928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.056879    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:49.057743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.057661    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-routing-overlay-volume\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:49.057882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.057782    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-plugins-config-volume\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:49.060181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.060157    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:49.066049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.066018    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttvb\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-kube-api-access-jttvb\") pod \"payload-processing-external-model-ff69f5485-2sfgv\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") " pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:49.214698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:49.214123    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:50.239156 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:41:50.239113    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae60b95e_6421_4598_943d_5089c12e193a.slice/crio-f4c3630e800e6544159806ac75dc7342e11339adaa8658516ffd22de9222f0d7 WatchSource:0}: Error finding container f4c3630e800e6544159806ac75dc7342e11339adaa8658516ffd22de9222f0d7: Status 404 returned error can't find the container with id f4c3630e800e6544159806ac75dc7342e11339adaa8658516ffd22de9222f0d7
Oct 30 21:41:50.240171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:50.240144    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"]
Oct 30 21:41:50.615767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:50.613084    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv" event={"ID":"ae60b95e-6421-4598-943d-5089c12e193a","Type":"ContainerStarted","Data":"9bd3c85e64b1f1427c2f821f122b0c3c0ab60e15646ed7c7f903975e892348a8"}
Oct 30 21:41:50.615767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:50.613151    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv" event={"ID":"ae60b95e-6421-4598-943d-5089c12e193a","Type":"ContainerStarted","Data":"f4c3630e800e6544159806ac75dc7342e11339adaa8658516ffd22de9222f0d7"}
Oct 30 21:41:50.615767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:50.613388    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:50.681892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:50.681830    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv" podStartSLOduration=2.681808072 podStartE2EDuration="2.681808072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:50.675977183 +0000 UTC m=+2419.025704591" watchObservedRunningTime="2026-10-30 21:41:50.681808072 +0000 UTC m=+2419.031535468"
Oct 30 21:41:50.702077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:50.702046    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"]
Oct 30 21:41:50.746268 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:50.746227    4633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca27eaecc19b: secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:50.746268 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:50.746263    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv: secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:50.746517 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:50.746346    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials podName:ae60b95e-6421-4598-943d-5089c12e193a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.246325874 +0000 UTC m=+2419.596053265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials") pod "payload-processing-external-model-ff69f5485-2sfgv" (UID: "ae60b95e-6421-4598-943d-5089c12e193a") : secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:51.253591 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:51.253554    4633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca27eaecc19b: secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:51.253591 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:51.253586    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv: secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:51.254175 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:51.253649    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials podName:ae60b95e-6421-4598-943d-5089c12e193a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:52.253631235 +0000 UTC m=+2420.603358608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials") pod "payload-processing-external-model-ff69f5485-2sfgv" (UID: "ae60b95e-6421-4598-943d-5089c12e193a") : secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:52.281151 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:52.281121    4633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca27eaecc19b: secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:52.281151 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:52.281151    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv: secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:52.281574 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:52.281207    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials podName:ae60b95e-6421-4598-943d-5089c12e193a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:54.281190872 +0000 UTC m=+2422.630918246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials") pod "payload-processing-external-model-ff69f5485-2sfgv" (UID: "ae60b95e-6421-4598-943d-5089c12e193a") : secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:54.344493 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:54.343252    4633 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ca27eaecc19b: secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:54.344493 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:54.343291    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv: secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:54.344493 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:54.343362    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials podName:ae60b95e-6421-4598-943d-5089c12e193a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:58.343342647 +0000 UTC m=+2426.693070021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials") pod "payload-processing-external-model-ff69f5485-2sfgv" (UID: "ae60b95e-6421-4598-943d-5089c12e193a") : secret "agc-cred-ca27eaecc19b" not found
Oct 30 21:41:54.503455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.503423    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"]
Oct 30 21:41:54.509115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.509087    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.516829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.516795    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"]
Oct 30 21:41:54.524041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.524013    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-plugins-e2e-extmod-b-36a099\""
Oct 30 21:41:54.524256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.524043    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-733bb027a41d\""
Oct 30 21:41:54.524376 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.524167    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"routing-overlay\""
Oct 30 21:41:54.524461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.524203    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"openshift-service-ca.crt\""
Oct 30 21:41:54.524461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.524330    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-e2e--3e617b48-dockercfg-5bf72\""
Oct 30 21:41:54.524578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.524433    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"kube-root-ca.crt\""
Oct 30 21:41:54.546624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.546595    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8w9\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-kube-api-access-5m8w9\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.546937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.546917    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.547174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.547128    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.547257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.547196    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.665611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.654107    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.665611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.654202    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8w9\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-kube-api-access-5m8w9\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.665611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.654506    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.665611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.654726    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.665611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.665342    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.728695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.715911    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.753697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.744744    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.762405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.762361    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8w9\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-kube-api-access-5m8w9\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:54.899718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:54.883818    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:55.364950 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:41:55.364853    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:41:55.367334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.367046    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"]
Oct 30 21:41:55.374861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.373908    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.378662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.378639    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"routing-overlay\""
Oct 30 21:41:55.378907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.378891    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"agc-cred-32727417a00a\""
Oct 30 21:41:55.379091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.379077    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"payload-processing-external-model-e2e--74dc6735-dockercfg-bqpjt\""
Oct 30 21:41:55.381544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.379847    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"kube-root-ca.crt\""
Oct 30 21:41:55.381762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.380161    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8072a0\""
Oct 30 21:41:55.381886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.380266    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"openshift-service-ca.crt\""
Oct 30 21:41:55.384175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.384147    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:41:55.386513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.386226    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"]
Oct 30 21:41:55.475121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.463488    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.475121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.463728    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpmrt\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-kube-api-access-vpmrt\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.475121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.463807    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.475121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.464013    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.738692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.731949    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.738692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.732113    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpmrt\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-kube-api-access-vpmrt\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.738692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.732187    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.738692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.732369    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.738692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.733501    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.900715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.900489    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpmrt\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-kube-api-access-vpmrt\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.936192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.935061    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:55.938218 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:41:55.938174    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82914d3_b295_49bd_8d44_4b5eec362ae3.slice/crio-e50a410df9f13d6c16ce55a9cc4db4420502f02bbef659012a4cd88ac1b1c2ac WatchSource:0}: Error finding container e50a410df9f13d6c16ce55a9cc4db4420502f02bbef659012a4cd88ac1b1c2ac: Status 404 returned error can't find the container with id e50a410df9f13d6c16ce55a9cc4db4420502f02bbef659012a4cd88ac1b1c2ac
Oct 30 21:41:55.944240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.943868    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"]
Oct 30 21:41:55.944359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:55.944251    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:56.035040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.035011    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:56.411344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.411317    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"]
Oct 30 21:41:56.411748 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:41:56.411368    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8775fd47_6920_4fc5_af37_8fc16679e970.slice/crio-e406509eefa6d442234a974ec611c61620e393f052cf8cefa2abbd4eb5dc8992 WatchSource:0}: Error finding container e406509eefa6d442234a974ec611c61620e393f052cf8cefa2abbd4eb5dc8992: Status 404 returned error can't find the container with id e406509eefa6d442234a974ec611c61620e393f052cf8cefa2abbd4eb5dc8992
Oct 30 21:41:56.943680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.943613    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4" event={"ID":"8775fd47-6920-4fc5-af37-8fc16679e970","Type":"ContainerStarted","Data":"2c124509ec2df94b52d7610ee3210eb49e038f6216139f95f32517accfff4728"}
Oct 30 21:41:56.944021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.943994    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4" event={"ID":"8775fd47-6920-4fc5-af37-8fc16679e970","Type":"ContainerStarted","Data":"e406509eefa6d442234a974ec611c61620e393f052cf8cefa2abbd4eb5dc8992"}
Oct 30 21:41:56.944189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.944164    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:41:56.945816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.945792    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88" event={"ID":"a82914d3-b295-49bd-8d44-4b5eec362ae3","Type":"ContainerStarted","Data":"7f43de400f408700618aea3bbe7a3e2f3a347c0922120776750e5888f8cc2830"}
Oct 30 21:41:56.945915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.945838    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88" event={"ID":"a82914d3-b295-49bd-8d44-4b5eec362ae3","Type":"ContainerStarted","Data":"e50a410df9f13d6c16ce55a9cc4db4420502f02bbef659012a4cd88ac1b1c2ac"}
Oct 30 21:41:56.945915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.945862    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:41:56.967420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.967374    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4" podStartSLOduration=1.967359583 podStartE2EDuration="1.967359583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:56.961105121 +0000 UTC m=+2425.310832553" watchObservedRunningTime="2026-10-30 21:41:56.967359583 +0000 UTC m=+2425.317086978"
Oct 30 21:41:56.982654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:56.982607    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88" podStartSLOduration=2.982594715 podStartE2EDuration="2.982594715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:56.977450692 +0000 UTC m=+2425.327178089" watchObservedRunningTime="2026-10-30 21:41:56.982594715 +0000 UTC m=+2425.332322116"
Oct 30 21:41:57.629753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:57.629710    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv" podUID="ae60b95e-6421-4598-943d-5089c12e193a" containerName="payload-processing" containerID="cri-o://9bd3c85e64b1f1427c2f821f122b0c3c0ab60e15646ed7c7f903975e892348a8" gracePeriod=25
Oct 30 21:41:57.959977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:57.959946    4633 generic.go:356] "Generic (PLEG): container finished" podID="ae60b95e-6421-4598-943d-5089c12e193a" containerID="9bd3c85e64b1f1427c2f821f122b0c3c0ab60e15646ed7c7f903975e892348a8" exitCode=0
Oct 30 21:41:57.960173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:57.960015    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv" event={"ID":"ae60b95e-6421-4598-943d-5089c12e193a","Type":"ContainerDied","Data":"9bd3c85e64b1f1427c2f821f122b0c3c0ab60e15646ed7c7f903975e892348a8"}
Oct 30 21:41:57.960173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:57.960074    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv" event={"ID":"ae60b95e-6421-4598-943d-5089c12e193a","Type":"ContainerDied","Data":"f4c3630e800e6544159806ac75dc7342e11339adaa8658516ffd22de9222f0d7"}
Oct 30 21:41:57.960173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:57.960091    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c3630e800e6544159806ac75dc7342e11339adaa8658516ffd22de9222f0d7"
Oct 30 21:41:57.980484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:57.980458    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:58.014850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.014814    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-plugins-config-volume\") pod \"ae60b95e-6421-4598-943d-5089c12e193a\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") "
Oct 30 21:41:58.016227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.016198    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-plugins-config-volume" pod "ae60b95e-6421-4598-943d-5089c12e193a" (UID: "ae60b95e-6421-4598-943d-5089c12e193a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:58.016227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.016209    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-routing-overlay-volume" pod "ae60b95e-6421-4598-943d-5089c12e193a" (UID: "ae60b95e-6421-4598-943d-5089c12e193a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:58.016457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.016326    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-routing-overlay-volume\") pod \"ae60b95e-6421-4598-943d-5089c12e193a\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") "
Oct 30 21:41:58.016457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.016382    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-kube-api-access-jttvb\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-kube-api-access-jttvb\") pod \"ae60b95e-6421-4598-943d-5089c12e193a\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") "
Oct 30 21:41:58.016576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.016527    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials\") pod \"ae60b95e-6421-4598-943d-5089c12e193a\" (UID: \"ae60b95e-6421-4598-943d-5089c12e193a\") "
Oct 30 21:41:58.019903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.019873    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:58.019903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.019900    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ae60b95e-6421-4598-943d-5089c12e193a-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:58.020078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.019987    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-kube-api-access-jttvb" pod "ae60b95e-6421-4598-943d-5089c12e193a" (UID: "ae60b95e-6421-4598-943d-5089c12e193a"). InnerVolumeSpecName "kube-api-access-jttvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:58.020300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.020272    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials" pod "ae60b95e-6421-4598-943d-5089c12e193a" (UID: "ae60b95e-6421-4598-943d-5089c12e193a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:58.124719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.124642    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jttvb\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-kube-api-access-jttvb\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:58.124719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.124702    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ae60b95e-6421-4598-943d-5089c12e193a-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:58.969864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:58.969830    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"
Oct 30 21:41:59.002081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:59.002035    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"]
Oct 30 21:41:59.008348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:59.008318    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff69f5485-2sfgv"]
Oct 30 21:41:59.827481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:41:59.827441    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae60b95e-6421-4598-943d-5089c12e193a" path="/var/lib/kubelet/pods/ae60b95e-6421-4598-943d-5089c12e193a/volumes"
Oct 30 21:42:07.963629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:07.963593    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:42:07.964360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:07.963699    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:42:19.196444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:19.196408    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:42:25.016976 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:42:25.016935    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:42:49.630712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:49.630484    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"]
Oct 30 21:42:49.631745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:49.631657    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy" containerID="cri-o://6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004" gracePeriod=30
Oct 30 21:42:54.959297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:54.959267    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:42:55.042855 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:42:55.042816    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:42:55.140131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140087    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-kube-api-access-4hxpt\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-kube-api-access-4hxpt\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.140351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140220    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-certs\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.140351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140316    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-data\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.140529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140510    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68456ea2-bbf0-467a-8c03-b371d72ac3be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68456ea2-bbf0-467a-8c03-b371d72ac3be-istiod-ca-cert\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.140611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140508    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-certs" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:55.140611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140556    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-socket\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.140611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140570    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-data" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:55.140859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140625    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-credential-socket\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.140859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140727    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-podinfo\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.140859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140778    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-envoy\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.140859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140821    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68456ea2-bbf0-467a-8c03-b371d72ac3be-istiod-ca-cert" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:55.140859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140828    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-token\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-token\") pod \"68456ea2-bbf0-467a-8c03-b371d72ac3be\" (UID: \"68456ea2-bbf0-467a-8c03-b371d72ac3be\") "
Oct 30 21:42:55.141148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140894    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-socket" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:55.141148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.140986    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-credential-socket" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:55.141641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.141613    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68456ea2-bbf0-467a-8c03-b371d72ac3be-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.141641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.141638    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.141937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.141648    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.141937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.141657    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.141937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.141683    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.142646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.142624    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-kube-api-access-4hxpt" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "kube-api-access-4hxpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:55.143080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.143063    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-podinfo" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:42:55.143602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.143571    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-envoy" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:55.143748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.143643    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-token" pod "68456ea2-bbf0-467a-8c03-b371d72ac3be" (UID: "68456ea2-bbf0-467a-8c03-b371d72ac3be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:55.243750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.243634    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.243750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.243686    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.243750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.243701    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.243750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.243722    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hxpt\" (UniqueName: \"kubernetes.io/projected/68456ea2-bbf0-467a-8c03-b371d72ac3be-kube-api-access-4hxpt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:55.271695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.266368    4633 generic.go:356] "Generic (PLEG): container finished" podID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerID="6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004" exitCode=0
Oct 30 21:42:55.271695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.266453    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"
Oct 30 21:42:55.271695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.266489    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" event={"ID":"68456ea2-bbf0-467a-8c03-b371d72ac3be","Type":"ContainerDied","Data":"6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004"}
Oct 30 21:42:55.271695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.266580    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk" event={"ID":"68456ea2-bbf0-467a-8c03-b371d72ac3be","Type":"ContainerDied","Data":"f143d6b232d4e10cdcf13ed3895da102b9447f9fe022bf94b028e81ab4ef2bd9"}
Oct 30 21:42:55.271695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.266605    4633 scope.go:122] "RemoveContainer" containerID="6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004"
Oct 30 21:42:55.293196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.293176    4633 scope.go:122] "RemoveContainer" containerID="6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004"
Oct 30 21:42:55.293476 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:42:55.293448    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004\": container with ID starting with 6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004 not found: ID does not exist" containerID="6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004"
Oct 30 21:42:55.293620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.293483    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004"} err="failed to get container status \"6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004\": rpc error: code = NotFound desc = could not find container \"6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004\": container with ID starting with 6bfc1e6aeb720f2586466bc2eb4b1e3ccd8b9f31eb6d9f6a7758b93e234f6004 not found: ID does not exist"
Oct 30 21:42:55.304512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.304489    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"]
Oct 30 21:42:55.310573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.310549    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-p6vvk"]
Oct 30 21:42:55.794979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:42:55.794949    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" path="/var/lib/kubelet/pods/68456ea2-bbf0-467a-8c03-b371d72ac3be/volumes"
Oct 30 21:43:06.909955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:06.909919    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:43:10.219056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:10.219020    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"]
Oct 30 21:43:10.219480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:10.219240    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy" containerID="cri-o://576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b" gracePeriod=30
Oct 30 21:43:15.522049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.521982    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:43:15.593779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.593745    4633 generic.go:356] "Generic (PLEG): container finished" podID="0ed24f34-9426-459c-8d08-28024297f237" containerID="576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b" exitCode=0
Oct 30 21:43:15.593967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.593798    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"
Oct 30 21:43:15.593967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.593837    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" event={"ID":"0ed24f34-9426-459c-8d08-28024297f237","Type":"ContainerDied","Data":"576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b"}
Oct 30 21:43:15.593967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.593938    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t" event={"ID":"0ed24f34-9426-459c-8d08-28024297f237","Type":"ContainerDied","Data":"0e4bf39fbe83d090f668618499812ff14cde693438d07861e5a49f6614f54b0c"}
Oct 30 21:43:15.594095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.593975    4633 scope.go:122] "RemoveContainer" containerID="576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b"
Oct 30 21:43:15.609045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.609019    4633 scope.go:122] "RemoveContainer" containerID="f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89"
Oct 30 21:43:15.622798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.622772    4633 scope.go:122] "RemoveContainer" containerID="576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b"
Oct 30 21:43:15.623186 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:43:15.623164    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b\": container with ID starting with 576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b not found: ID does not exist" containerID="576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b"
Oct 30 21:43:15.623277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.623198    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b"} err="failed to get container status \"576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b\": rpc error: code = NotFound desc = could not find container \"576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b\": container with ID starting with 576cc43482d340ea3078c106d46a0d9e74f372a8237886300f439cd123bf001b not found: ID does not exist"
Oct 30 21:43:15.623277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.623224    4633 scope.go:122] "RemoveContainer" containerID="f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89"
Oct 30 21:43:15.623543 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:43:15.623521    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89\": container with ID starting with f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89 not found: ID does not exist" containerID="f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89"
Oct 30 21:43:15.623593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.623554    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89"} err="failed to get container status \"f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89\": rpc error: code = NotFound desc = could not find container \"f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89\": container with ID starting with f8364902200723f3363c097c5caeab82c99ecce6dc2f1df7f0cd0fe0da56cc89 not found: ID does not exist"
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.722056    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-data\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.722210    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ed24f34-9426-459c-8d08-28024297f237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed24f34-9426-459c-8d08-28024297f237-istio-podinfo\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.722355    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-istio-token\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-istio-token\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.722511    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-credential-socket\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.722635    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-kube-api-access-m9xk6\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-kube-api-access-m9xk6\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.722847    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-socket\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.722910    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-envoy\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.722995    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed24f34-9426-459c-8d08-28024297f237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed24f34-9426-459c-8d08-28024297f237-istiod-ca-cert\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.724094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.723026    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-certs\") pod \"0ed24f34-9426-459c-8d08-28024297f237\" (UID: \"0ed24f34-9426-459c-8d08-28024297f237\") "
Oct 30 21:43:15.728223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.725060    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-certs" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:15.729188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.728775    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-data" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:15.730266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.730188    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-socket" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:15.731082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.730940    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-credential-socket" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:15.731082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.731044    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed24f34-9426-459c-8d08-28024297f237-istiod-ca-cert" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:15.734813 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.734752    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ed24f34-9426-459c-8d08-28024297f237-istio-podinfo" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:43:15.736558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.736528    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-envoy" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:15.736703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.736571    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-istio-token" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:15.738070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.738042    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-kube-api-access-m9xk6" pod "0ed24f34-9426-459c-8d08-28024297f237" (UID: "0ed24f34-9426-459c-8d08-28024297f237"). InnerVolumeSpecName "kube-api-access-m9xk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:15.826365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826333    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.826365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826363    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.826365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826375    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9xk6\" (UniqueName: \"kubernetes.io/projected/0ed24f34-9426-459c-8d08-28024297f237-kube-api-access-m9xk6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.826646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826385    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.826646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826397    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.826646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826406    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed24f34-9426-459c-8d08-28024297f237-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.826646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826414    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.826646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826422    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed24f34-9426-459c-8d08-28024297f237-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.826646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.826432    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed24f34-9426-459c-8d08-28024297f237-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:15.915391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.915354    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"]
Oct 30 21:43:15.921102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:15.921071    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-9bc5t"]
Oct 30 21:43:17.850293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:17.850253    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed24f34-9426-459c-8d08-28024297f237" path="/var/lib/kubelet/pods/0ed24f34-9426-459c-8d08-28024297f237/volumes"
Oct 30 21:43:25.018983 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:43:25.018937    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:43:53.443657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:43:53.443608    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:43:55.017581 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:43:55.017535    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:44:25.086199 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:44:25.086089    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:44:28.059776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:28.059739    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:44:32.980892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.980856    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"]
Oct 30 21:44:32.982139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982108    4633 cpu_manager.go:414] "containerMap: removing container" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy"
Oct 30 21:44:32.982139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982139    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy"
Oct 30 21:44:32.982364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982189    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ae60b95e-6421-4598-943d-5089c12e193a" containerName="payload-processing"
Oct 30 21:44:32.982364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982197    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae60b95e-6421-4598-943d-5089c12e193a" containerName="payload-processing"
Oct 30 21:44:32.982364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982231    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy"
Oct 30 21:44:32.982364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982239    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy"
Oct 30 21:44:32.982364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982252    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy"
Oct 30 21:44:32.982364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982259    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy"
Oct 30 21:44:32.982733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982452    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy"
Oct 30 21:44:32.982733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982473    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="68456ea2-bbf0-467a-8c03-b371d72ac3be" containerName="istio-proxy"
Oct 30 21:44:32.982733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.982489    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae60b95e-6421-4598-943d-5089c12e193a" containerName="payload-processing"
Oct 30 21:44:32.999859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.992863    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"]
Oct 30 21:44:32.999859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:32.993019    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.006838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.006803    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"agc-cred-13a652de1717\""
Oct 30 21:44:33.097510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.097475    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmd8\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-kube-api-access-gvmd8\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.097735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.097677    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.097827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.097815    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.097958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.097943    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.199472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.199438    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmd8\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-kube-api-access-gvmd8\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.199713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.199577    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.199798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.199754    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.199916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.199901    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.200556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.200532    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.200638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.200591    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.202932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.202906    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.211693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.211648    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmd8\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-kube-api-access-gvmd8\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.320724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.320653    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:33.625089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.625053    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"]
Oct 30 21:44:33.643141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:33.643108    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff" event={"ID":"1fe55c35-e204-4e19-8a12-0f2ece5d70c8","Type":"ContainerStarted","Data":"c6f9560882ef9a35027e26701719bcd71a9f94e8e5658b38d514413ca2c74163"}
Oct 30 21:44:34.668464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:34.668418    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff" event={"ID":"1fe55c35-e204-4e19-8a12-0f2ece5d70c8","Type":"ContainerStarted","Data":"a0da16d3e350b0cc95c0b82200e06cf52a1420aa2c2cbceecc0a1cb1a8a36320"}
Oct 30 21:44:34.668954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:34.668636    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:34.695784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:34.695727    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff" podStartSLOduration=2.695712834 podStartE2EDuration="2.695712834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:34.689771279 +0000 UTC m=+2583.039498674" watchObservedRunningTime="2026-10-30 21:44:34.695712834 +0000 UTC m=+2583.045440207"
Oct 30 21:44:37.507624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:37.507584    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"]
Oct 30 21:44:42.508832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.508792    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g" podUID="a4de7390-e80b-4c49-84a8-b178ecb8a27d" containerName="payload-processing" containerID="cri-o://ab4753438ab9c9c12d856b642a06e90cc660421573f247653e6fccf487f17ae7" gracePeriod=25
Oct 30 21:44:42.748899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.748860    4633 generic.go:356] "Generic (PLEG): container finished" podID="a4de7390-e80b-4c49-84a8-b178ecb8a27d" containerID="ab4753438ab9c9c12d856b642a06e90cc660421573f247653e6fccf487f17ae7" exitCode=0
Oct 30 21:44:42.749197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.749176    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g" event={"ID":"a4de7390-e80b-4c49-84a8-b178ecb8a27d","Type":"ContainerDied","Data":"ab4753438ab9c9c12d856b642a06e90cc660421573f247653e6fccf487f17ae7"}
Oct 30 21:44:42.818450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.818417    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:44:42.908194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.908160    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-routing-overlay-volume\") pod \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") "
Oct 30 21:44:42.908421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.908253    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-kube-api-access-x29dt\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-kube-api-access-x29dt\") pod \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") "
Oct 30 21:44:42.908421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.908283    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-provider-credentials\") pod \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") "
Oct 30 21:44:42.908638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.908615    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-routing-overlay-volume" pod "a4de7390-e80b-4c49-84a8-b178ecb8a27d" (UID: "a4de7390-e80b-4c49-84a8-b178ecb8a27d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:42.908758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.908627    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-plugins-config-volume\") pod \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\" (UID: \"a4de7390-e80b-4c49-84a8-b178ecb8a27d\") "
Oct 30 21:44:42.909018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.908997    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-plugins-config-volume" pod "a4de7390-e80b-4c49-84a8-b178ecb8a27d" (UID: "a4de7390-e80b-4c49-84a8-b178ecb8a27d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:42.909856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.909786    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:42.909856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.909817    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4de7390-e80b-4c49-84a8-b178ecb8a27d-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:42.911267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.911247    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-kube-api-access-x29dt" pod "a4de7390-e80b-4c49-84a8-b178ecb8a27d" (UID: "a4de7390-e80b-4c49-84a8-b178ecb8a27d"). InnerVolumeSpecName "kube-api-access-x29dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:42.911363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:42.911282    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-provider-credentials" pod "a4de7390-e80b-4c49-84a8-b178ecb8a27d" (UID: "a4de7390-e80b-4c49-84a8-b178ecb8a27d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:43.011491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:43.011454    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x29dt\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-kube-api-access-x29dt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.011491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:43.011487    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4de7390-e80b-4c49-84a8-b178ecb8a27d-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.758900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:43.758867    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"
Oct 30 21:44:43.759405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:43.758904    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g" event={"ID":"a4de7390-e80b-4c49-84a8-b178ecb8a27d","Type":"ContainerDied","Data":"4f9ab60f40faaad2e54adcbeda7c1ff7c5058c0473c312d50b76520db9f5b18f"}
Oct 30 21:44:43.759405 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:43.758962    4633 scope.go:122] "RemoveContainer" containerID="ab4753438ab9c9c12d856b642a06e90cc660421573f247653e6fccf487f17ae7"
Oct 30 21:44:43.806338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:43.806303    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"]
Oct 30 21:44:43.806338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:43.806345    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-bf55clg7g"]
Oct 30 21:44:45.573023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:45.572983    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"]
Oct 30 21:44:45.682323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:45.682293    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:44:45.764865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:45.764826    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"]
Oct 30 21:44:45.796195 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:45.796156    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4de7390-e80b-4c49-84a8-b178ecb8a27d" path="/var/lib/kubelet/pods/a4de7390-e80b-4c49-84a8-b178ecb8a27d/volumes"
Oct 30 21:44:49.004814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.004784    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"]
Oct 30 21:44:49.005587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.005573    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a4de7390-e80b-4c49-84a8-b178ecb8a27d" containerName="payload-processing"
Oct 30 21:44:49.005649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.005589    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4de7390-e80b-4c49-84a8-b178ecb8a27d" containerName="payload-processing"
Oct 30 21:44:49.005724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.005714    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4de7390-e80b-4c49-84a8-b178ecb8a27d" containerName="payload-processing"
Oct 30 21:44:49.005776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.005732    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed24f34-9426-459c-8d08-28024297f237" containerName="istio-proxy"
Oct 30 21:44:49.011150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.011128    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.026685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.026636    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"]
Oct 30 21:44:49.032996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.032968    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e596df\""
Oct 30 21:44:49.033264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.032976    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"kube-root-ca.crt\""
Oct 30 21:44:49.033264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.033041    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"openshift-service-ca.crt\""
Oct 30 21:44:49.033477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.033101    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"agc-cred-0e714968df36\""
Oct 30 21:44:49.033477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.033167    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"routing-overlay\""
Oct 30 21:44:49.228704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.207575    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.229162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.229132    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.229501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.229480    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.251695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.229981    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dpt\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-kube-api-access-w4dpt\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.353880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.353777    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.371914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.371868    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.396696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.387793    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.406817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.402211    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dpt\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-kube-api-access-w4dpt\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.451899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.386747    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.497094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.397918    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.497389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.415481    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.497540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.458371    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dpt\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-kube-api-access-w4dpt\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:49.629341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:49.629253    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:50.188098 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:44:50.188023    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ec9a01_f868_4b7f_ba04_40a340ad26b2.slice/crio-0333b84401f389f37964800d46ac72ee10cdf70313b0efad9288461962b6cc05 WatchSource:0}: Error finding container 0333b84401f389f37964800d46ac72ee10cdf70313b0efad9288461962b6cc05: Status 404 returned error can't find the container with id 0333b84401f389f37964800d46ac72ee10cdf70313b0efad9288461962b6cc05
Oct 30 21:44:50.189450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.189417    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"]
Oct 30 21:44:50.587762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.587482    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88" podUID="a82914d3-b295-49bd-8d44-4b5eec362ae3" containerName="payload-processing" containerID="cri-o://7f43de400f408700618aea3bbe7a3e2f3a347c0922120776750e5888f8cc2830" gracePeriod=25
Oct 30 21:44:50.765760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.765710    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4" podUID="8775fd47-6920-4fc5-af37-8fc16679e970" containerName="payload-processing" containerID="cri-o://2c124509ec2df94b52d7610ee3210eb49e038f6216139f95f32517accfff4728" gracePeriod=25
Oct 30 21:44:50.931249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.931215    4633 generic.go:356] "Generic (PLEG): container finished" podID="a82914d3-b295-49bd-8d44-4b5eec362ae3" containerID="7f43de400f408700618aea3bbe7a3e2f3a347c0922120776750e5888f8cc2830" exitCode=0
Oct 30 21:44:50.931423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.931341    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88" event={"ID":"a82914d3-b295-49bd-8d44-4b5eec362ae3","Type":"ContainerDied","Data":"7f43de400f408700618aea3bbe7a3e2f3a347c0922120776750e5888f8cc2830"}
Oct 30 21:44:50.931423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.931390    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88" event={"ID":"a82914d3-b295-49bd-8d44-4b5eec362ae3","Type":"ContainerDied","Data":"e50a410df9f13d6c16ce55a9cc4db4420502f02bbef659012a4cd88ac1b1c2ac"}
Oct 30 21:44:50.931423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.931413    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e50a410df9f13d6c16ce55a9cc4db4420502f02bbef659012a4cd88ac1b1c2ac"
Oct 30 21:44:50.935897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.933636    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk" event={"ID":"b6ec9a01-f868-4b7f-ba04-40a340ad26b2","Type":"ContainerStarted","Data":"6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1"}
Oct 30 21:44:50.935897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.933728    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk" event={"ID":"b6ec9a01-f868-4b7f-ba04-40a340ad26b2","Type":"ContainerStarted","Data":"0333b84401f389f37964800d46ac72ee10cdf70313b0efad9288461962b6cc05"}
Oct 30 21:44:50.935897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.934830    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:44:50.936850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.936827    4633 generic.go:356] "Generic (PLEG): container finished" podID="8775fd47-6920-4fc5-af37-8fc16679e970" containerID="2c124509ec2df94b52d7610ee3210eb49e038f6216139f95f32517accfff4728" exitCode=0
Oct 30 21:44:50.937059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.937043    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4" event={"ID":"8775fd47-6920-4fc5-af37-8fc16679e970","Type":"ContainerDied","Data":"2c124509ec2df94b52d7610ee3210eb49e038f6216139f95f32517accfff4728"}
Oct 30 21:44:50.952768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.946777    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:44:50.980875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:50.979928    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk" podStartSLOduration=2.9799087269999998 podStartE2EDuration="2.979908727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:50.970125776 +0000 UTC m=+2599.319853171" watchObservedRunningTime="2026-10-30 21:44:50.979908727 +0000 UTC m=+2599.329636124"
Oct 30 21:44:51.021531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.021493    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-kube-api-access-5m8w9\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-kube-api-access-5m8w9\") pod \"a82914d3-b295-49bd-8d44-4b5eec362ae3\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") "
Oct 30 21:44:51.021953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.021928    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-routing-overlay-volume\") pod \"a82914d3-b295-49bd-8d44-4b5eec362ae3\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") "
Oct 30 21:44:51.022115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.022058    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-provider-credentials\") pod \"a82914d3-b295-49bd-8d44-4b5eec362ae3\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") "
Oct 30 21:44:51.022115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.022107    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-plugins-config-volume\") pod \"a82914d3-b295-49bd-8d44-4b5eec362ae3\" (UID: \"a82914d3-b295-49bd-8d44-4b5eec362ae3\") "
Oct 30 21:44:51.022981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.022952    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-routing-overlay-volume" pod "a82914d3-b295-49bd-8d44-4b5eec362ae3" (UID: "a82914d3-b295-49bd-8d44-4b5eec362ae3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:51.024884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.024098    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-plugins-config-volume" pod "a82914d3-b295-49bd-8d44-4b5eec362ae3" (UID: "a82914d3-b295-49bd-8d44-4b5eec362ae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:51.026372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.026338    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-provider-credentials" pod "a82914d3-b295-49bd-8d44-4b5eec362ae3" (UID: "a82914d3-b295-49bd-8d44-4b5eec362ae3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:51.028719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.028646    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-kube-api-access-5m8w9" pod "a82914d3-b295-49bd-8d44-4b5eec362ae3" (UID: "a82914d3-b295-49bd-8d44-4b5eec362ae3"). InnerVolumeSpecName "kube-api-access-5m8w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:51.109347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.109320    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:44:51.125882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.125808    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m8w9\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-kube-api-access-5m8w9\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.125882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.125834    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.125882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.125845    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a82914d3-b295-49bd-8d44-4b5eec362ae3-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.125882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.125856    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82914d3-b295-49bd-8d44-4b5eec362ae3-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.226555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.226511    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-routing-overlay-volume\") pod \"8775fd47-6920-4fc5-af37-8fc16679e970\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") "
Oct 30 21:44:51.227156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.226604    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-kube-api-access-vpmrt\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-kube-api-access-vpmrt\") pod \"8775fd47-6920-4fc5-af37-8fc16679e970\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") "
Oct 30 21:44:51.227156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.226656    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-provider-credentials\") pod \"8775fd47-6920-4fc5-af37-8fc16679e970\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") "
Oct 30 21:44:51.227156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.227010    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-plugins-config-volume\") pod \"8775fd47-6920-4fc5-af37-8fc16679e970\" (UID: \"8775fd47-6920-4fc5-af37-8fc16679e970\") "
Oct 30 21:44:51.227156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.227034    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-routing-overlay-volume" pod "8775fd47-6920-4fc5-af37-8fc16679e970" (UID: "8775fd47-6920-4fc5-af37-8fc16679e970"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:51.227913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.227885    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.228087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.228062    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-plugins-config-volume" pod "8775fd47-6920-4fc5-af37-8fc16679e970" (UID: "8775fd47-6920-4fc5-af37-8fc16679e970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:51.230230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.230201    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-kube-api-access-vpmrt" pod "8775fd47-6920-4fc5-af37-8fc16679e970" (UID: "8775fd47-6920-4fc5-af37-8fc16679e970"). InnerVolumeSpecName "kube-api-access-vpmrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:51.232556 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.232521    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-provider-credentials" pod "8775fd47-6920-4fc5-af37-8fc16679e970" (UID: "8775fd47-6920-4fc5-af37-8fc16679e970"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:51.333697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.332258    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8775fd47-6920-4fc5-af37-8fc16679e970-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.333697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.332299    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpmrt\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-kube-api-access-vpmrt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.333697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.332313    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8775fd47-6920-4fc5-af37-8fc16679e970-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.628259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.599248    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"]
Oct 30 21:44:51.703757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.633145    4633 cpu_manager.go:414] "containerMap: removing container" podUID="8775fd47-6920-4fc5-af37-8fc16679e970" containerName="payload-processing"
Oct 30 21:44:51.704742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.704710    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8775fd47-6920-4fc5-af37-8fc16679e970" containerName="payload-processing"
Oct 30 21:44:51.704984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.704967    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a82914d3-b295-49bd-8d44-4b5eec362ae3" containerName="payload-processing"
Oct 30 21:44:51.705191 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.705172    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82914d3-b295-49bd-8d44-4b5eec362ae3" containerName="payload-processing"
Oct 30 21:44:51.705555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.705535    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8775fd47-6920-4fc5-af37-8fc16679e970" containerName="payload-processing"
Oct 30 21:44:51.705725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:51.705684    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82914d3-b295-49bd-8d44-4b5eec362ae3" containerName="payload-processing"
Oct 30 21:44:52.700690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.689991    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.700690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.690127    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.700690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.690257    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.700690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.690363    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96b8z\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-kube-api-access-96b8z\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.712258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.712223    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"
Oct 30 21:44:52.753605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.720808    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"
Oct 30 21:44:52.754044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.754015    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.798446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.778321    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"openshift-service-ca.crt\""
Oct 30 21:44:52.798446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.778456    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"routing-overlay\""
Oct 30 21:44:52.798446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.778572    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-e2e--650b641a-dockercfg-r6s2h\""
Oct 30 21:44:52.798446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.786976    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-bb672c18dcea\""
Oct 30 21:44:52.798446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.787097    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1ed97\""
Oct 30 21:44:52.798446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.787163    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"kube-root-ca.crt\""
Oct 30 21:44:52.819193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.819147    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.819567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.819548    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.819931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.819912    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.820225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.820207    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96b8z\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-kube-api-access-96b8z\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.830091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.825786    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.830091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.826473    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.942695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.933688    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96b8z\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-kube-api-access-96b8z\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:52.965748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:52.957434    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:53.015112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.013101    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:44:53.038268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.038228    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"]
Oct 30 21:44:53.038575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.038546    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4" event={"ID":"8775fd47-6920-4fc5-af37-8fc16679e970","Type":"ContainerDied","Data":"e406509eefa6d442234a974ec611c61620e393f052cf8cefa2abbd4eb5dc8992"}
Oct 30 21:44:53.040350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.040333    4633 scope.go:122] "RemoveContainer" containerID="2c124509ec2df94b52d7610ee3210eb49e038f6216139f95f32517accfff4728"
Oct 30 21:44:53.260003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.221781    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"]
Oct 30 21:44:53.291863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.291834    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:53.399479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.384882    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bch9tj4"]
Oct 30 21:44:53.502695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.500751    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"]
Oct 30 21:44:53.614380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.614282    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-67c7jgd88"]
Oct 30 21:44:53.859693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.858812    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8775fd47-6920-4fc5-af37-8fc16679e970" path="/var/lib/kubelet/pods/8775fd47-6920-4fc5-af37-8fc16679e970/volumes"
Oct 30 21:44:53.859693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:53.859480    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82914d3-b295-49bd-8d44-4b5eec362ae3" path="/var/lib/kubelet/pods/a82914d3-b295-49bd-8d44-4b5eec362ae3/volumes"
Oct 30 21:44:54.292874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:54.283153    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"]
Oct 30 21:44:54.607902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:54.607858    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" event={"ID":"972f8efc-fe31-4e9f-8f39-09725ca385fa","Type":"ContainerStarted","Data":"ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587"}
Oct 30 21:44:54.608107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:54.607910    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" event={"ID":"972f8efc-fe31-4e9f-8f39-09725ca385fa","Type":"ContainerStarted","Data":"14f66d162ee13d002ea3ad86e490aa905a915dda4c95e908b6744959e076f625"}
Oct 30 21:44:54.608107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:54.607946    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:44:54.633586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:54.633531    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" podStartSLOduration=3.633514375 podStartE2EDuration="3.633514375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:54.629906953 +0000 UTC m=+2602.979634372" watchObservedRunningTime="2026-10-30 21:44:54.633514375 +0000 UTC m=+2602.983241789"
Oct 30 21:44:55.152698 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:44:55.146541    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:44:55.421942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.421857    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"]
Oct 30 21:44:55.427902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.427876    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.432286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.432262    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"kube-root-ca.crt\""
Oct 30 21:44:55.432286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.432280    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"openshift-service-ca.crt\""
Oct 30 21:44:55.432529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.432513    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"payload-processing-external-model-plugins-e2e-extmod-a-fb2b58\""
Oct 30 21:44:55.433636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.433218    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"routing-overlay\""
Oct 30 21:44:55.440865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.440830    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"agc-cred-98afef9e5c37\""
Oct 30 21:44:55.441506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.441482    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"]
Oct 30 21:44:55.445724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.445682    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"payload-processing-external-model-e2e--b5c8139c-dockercfg-b6q5w\""
Oct 30 21:44:55.524226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.524194    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js6ld\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-kube-api-access-js6ld\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.524410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.524256    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.524453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.524404    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.524501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.524487    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.626017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.625983    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.626217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.626166    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js6ld\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-kube-api-access-js6ld\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.626217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.626209    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.626350 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.626286    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.626762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.626739    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.627055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.627037    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.629526 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.629501    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.642196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.642148    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js6ld\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-kube-api-access-js6ld\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:55.756753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:55.756280    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:56.349518 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:44:56.349489    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2796e7a_8051_4b7c_80ac_14d63c153691.slice/crio-c798a583ec54b6bdc4b522a739cf7b9549ed2b897c791e1c18fce966c76ca88b WatchSource:0}: Error finding container c798a583ec54b6bdc4b522a739cf7b9549ed2b897c791e1c18fce966c76ca88b: Status 404 returned error can't find the container with id c798a583ec54b6bdc4b522a739cf7b9549ed2b897c791e1c18fce966c76ca88b
Oct 30 21:44:56.349996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:56.349975    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"]
Oct 30 21:44:56.627875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:56.627781    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr" event={"ID":"c2796e7a-8051-4b7c-80ac-14d63c153691","Type":"ContainerStarted","Data":"5022ee12c2d292e061238e2fe98a0dc3065a6d739a664418018fe94dabc40876"}
Oct 30 21:44:56.627875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:56.627836    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr" event={"ID":"c2796e7a-8051-4b7c-80ac-14d63c153691","Type":"ContainerStarted","Data":"c798a583ec54b6bdc4b522a739cf7b9549ed2b897c791e1c18fce966c76ca88b"}
Oct 30 21:44:56.627875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:56.627856    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:44:56.664365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:56.664315    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr" podStartSLOduration=1.66429992 podStartE2EDuration="1.66429992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:56.661990189 +0000 UTC m=+2605.011717585" watchObservedRunningTime="2026-10-30 21:44:56.66429992 +0000 UTC m=+2605.014027306"
Oct 30 21:44:59.539099 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.539070    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"]
Oct 30 21:44:59.543997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.543962    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:44:59.554074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.554049    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-0230a6\"/\"kube-root-ca.crt\""
Oct 30 21:44:59.555631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.555432    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-0230a6\"/\"openshift-service-ca.crt\""
Oct 30 21:44:59.555631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.555596    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-0230a6\"/\"default-dockercfg-9slhd\""
Oct 30 21:44:59.556204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.556182    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-0230a6\"/\"e2e-usage-sim-e580cf-cert\""
Oct 30 21:44:59.560777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.560749    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"]
Oct 30 21:44:59.570541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.570500    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvggz\" (UniqueName: \"kubernetes.io/projected/18a7653d-8782-4705-b398-e89d2d9d1fb2-kube-api-access-hvggz\") pod \"e2e-usage-sim-e580cf-54cdc9877d-mqngt\" (UID: \"18a7653d-8782-4705-b398-e89d2d9d1fb2\") " pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:44:59.570733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.570703    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18a7653d-8782-4705-b398-e89d2d9d1fb2-tls\") pod \"e2e-usage-sim-e580cf-54cdc9877d-mqngt\" (UID: \"18a7653d-8782-4705-b398-e89d2d9d1fb2\") " pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:44:59.672593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.672556    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvggz\" (UniqueName: \"kubernetes.io/projected/18a7653d-8782-4705-b398-e89d2d9d1fb2-kube-api-access-hvggz\") pod \"e2e-usage-sim-e580cf-54cdc9877d-mqngt\" (UID: \"18a7653d-8782-4705-b398-e89d2d9d1fb2\") " pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:44:59.672821 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.672662    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18a7653d-8782-4705-b398-e89d2d9d1fb2-tls\") pod \"e2e-usage-sim-e580cf-54cdc9877d-mqngt\" (UID: \"18a7653d-8782-4705-b398-e89d2d9d1fb2\") " pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:44:59.675567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.675543    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18a7653d-8782-4705-b398-e89d2d9d1fb2-tls\") pod \"e2e-usage-sim-e580cf-54cdc9877d-mqngt\" (UID: \"18a7653d-8782-4705-b398-e89d2d9d1fb2\") " pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:44:59.681185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.681159    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvggz\" (UniqueName: \"kubernetes.io/projected/18a7653d-8782-4705-b398-e89d2d9d1fb2-kube-api-access-hvggz\") pod \"e2e-usage-sim-e580cf-54cdc9877d-mqngt\" (UID: \"18a7653d-8782-4705-b398-e89d2d9d1fb2\") " pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:44:59.859739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:44:59.859603    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:45:00.228022 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:45:00.227986    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18a7653d_8782_4705_b398_e89d2d9d1fb2.slice/crio-0c4bbf11f165ad927afd05c02696cb4457015d73e74e46f570dd3db7e66c42bc WatchSource:0}: Error finding container 0c4bbf11f165ad927afd05c02696cb4457015d73e74e46f570dd3db7e66c42bc: Status 404 returned error can't find the container with id 0c4bbf11f165ad927afd05c02696cb4457015d73e74e46f570dd3db7e66c42bc
Oct 30 21:45:00.230419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.230389    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"]
Oct 30 21:45:00.382220 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.382178    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"]
Oct 30 21:45:00.387458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.387436    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:00.398111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.398085    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"]
Oct 30 21:45:00.412551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.412513    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:45:00.445187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.445157    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"]
Oct 30 21:45:00.450426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.450401    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:00.466300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.466267    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"]
Oct 30 21:45:00.480049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.479215    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:00.490576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.490548    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"]
Oct 30 21:45:00.537762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.535877    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9\" (UID: \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:00.537762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.535976    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2tw\" (UniqueName: \"kubernetes.io/projected/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-kube-api-access-kg2tw\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9\" (UID: \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:00.537762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.536131    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxj5\" (UniqueName: \"kubernetes.io/projected/eb72426b-4170-4589-af8d-4f117cb92f2c-kube-api-access-zvxj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66\" (UID: \"eb72426b-4170-4589-af8d-4f117cb92f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:00.537762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.536192    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72426b-4170-4589-af8d-4f117cb92f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66\" (UID: \"eb72426b-4170-4589-af8d-4f117cb92f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:00.550442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.550406    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"]
Oct 30 21:45:00.557565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.557534    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:00.565176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.564697    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"]
Oct 30 21:45:00.576789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.576760    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"]
Oct 30 21:45:00.637316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.637266    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8fr\" (UniqueName: \"kubernetes.io/projected/c4815255-f0a6-41c8-801b-17f2b1ff88f1-kube-api-access-hd8fr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj\" (UID: \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:00.637316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.637313    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxj5\" (UniqueName: \"kubernetes.io/projected/eb72426b-4170-4589-af8d-4f117cb92f2c-kube-api-access-zvxj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66\" (UID: \"eb72426b-4170-4589-af8d-4f117cb92f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:00.637608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.637427    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72426b-4170-4589-af8d-4f117cb92f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66\" (UID: \"eb72426b-4170-4589-af8d-4f117cb92f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:00.637608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.637565    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4815255-f0a6-41c8-801b-17f2b1ff88f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj\" (UID: \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:00.637753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.637618    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wnq\" (UniqueName: \"kubernetes.io/projected/0495337a-833d-49a6-ab96-e0a93995d335-kube-api-access-p5wnq\") pod \"maas-api-key-cleanup-29889945-twf4t\" (UID: \"0495337a-833d-49a6-ab96-e0a93995d335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:00.637753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.637715    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9\" (UID: \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:00.637753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.637750    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0495337a-833d-49a6-ab96-e0a93995d335-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-twf4t\" (UID: \"0495337a-833d-49a6-ab96-e0a93995d335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:00.637933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.637784    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2tw\" (UniqueName: \"kubernetes.io/projected/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-kube-api-access-kg2tw\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9\" (UID: \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:00.638225 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.638204    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72426b-4170-4589-af8d-4f117cb92f2c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66\" (UID: \"eb72426b-4170-4589-af8d-4f117cb92f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:00.638332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.638313    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9\" (UID: \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:00.647697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.647659    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxj5\" (UniqueName: \"kubernetes.io/projected/eb72426b-4170-4589-af8d-4f117cb92f2c-kube-api-access-zvxj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66\" (UID: \"eb72426b-4170-4589-af8d-4f117cb92f2c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:00.661242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.661218    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2tw\" (UniqueName: \"kubernetes.io/projected/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-kube-api-access-kg2tw\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9\" (UID: \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:00.707116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.707081    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:00.739791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.739723    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4815255-f0a6-41c8-801b-17f2b1ff88f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj\" (UID: \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:00.739941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.739803    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wnq\" (UniqueName: \"kubernetes.io/projected/0495337a-833d-49a6-ab96-e0a93995d335-kube-api-access-p5wnq\") pod \"maas-api-key-cleanup-29889945-twf4t\" (UID: \"0495337a-833d-49a6-ab96-e0a93995d335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:00.739941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.739858    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0495337a-833d-49a6-ab96-e0a93995d335-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-twf4t\" (UID: \"0495337a-833d-49a6-ab96-e0a93995d335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:00.740074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.739998    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8fr\" (UniqueName: \"kubernetes.io/projected/c4815255-f0a6-41c8-801b-17f2b1ff88f1-kube-api-access-hd8fr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj\" (UID: \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:00.740501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.740476    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4815255-f0a6-41c8-801b-17f2b1ff88f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj\" (UID: \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:00.740662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.740643    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0495337a-833d-49a6-ab96-e0a93995d335-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-twf4t\" (UID: \"0495337a-833d-49a6-ab96-e0a93995d335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:00.758797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.758759    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8fr\" (UniqueName: \"kubernetes.io/projected/c4815255-f0a6-41c8-801b-17f2b1ff88f1-kube-api-access-hd8fr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj\" (UID: \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:00.758983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.758963    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wnq\" (UniqueName: \"kubernetes.io/projected/0495337a-833d-49a6-ab96-e0a93995d335-kube-api-access-p5wnq\") pod \"maas-api-key-cleanup-29889945-twf4t\" (UID: \"0495337a-833d-49a6-ab96-e0a93995d335\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:00.793830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.793795    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:00.807512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.807482    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:00.882720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.882377    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:00.937367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.937333    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt" event={"ID":"18a7653d-8782-4705-b398-e89d2d9d1fb2","Type":"ContainerStarted","Data":"ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f"}
Oct 30 21:45:00.937617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.937376    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt" event={"ID":"18a7653d-8782-4705-b398-e89d2d9d1fb2","Type":"ContainerStarted","Data":"0c4bbf11f165ad927afd05c02696cb4457015d73e74e46f570dd3db7e66c42bc"}
Oct 30 21:45:00.937822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.937797    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:45:00.942973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.942944    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:45:00.979327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:00.979266    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt" podStartSLOduration=1.979248176 podStartE2EDuration="1.979248176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:00.977402682 +0000 UTC m=+2609.327130079" watchObservedRunningTime="2026-10-30 21:45:00.979248176 +0000 UTC m=+2609.328975575"
Oct 30 21:45:01.021559 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:45:01.017106    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb72426b_4170_4589_af8d_4f117cb92f2c.slice/crio-f03d616aac9759e4655a041228e87b316e6a7f88af21ac8e6075404785198300 WatchSource:0}: Error finding container f03d616aac9759e4655a041228e87b316e6a7f88af21ac8e6075404785198300: Status 404 returned error can't find the container with id f03d616aac9759e4655a041228e87b316e6a7f88af21ac8e6075404785198300
Oct 30 21:45:01.023963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.023940    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"]
Oct 30 21:45:01.228474 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:45:01.228407    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4815255_f0a6_41c8_801b_17f2b1ff88f1.slice/crio-2f41ead709d76f93c5cd4f5280c8024933e5b488ef52196fd1ffe3d16fe4b8a0 WatchSource:0}: Error finding container 2f41ead709d76f93c5cd4f5280c8024933e5b488ef52196fd1ffe3d16fe4b8a0: Status 404 returned error can't find the container with id 2f41ead709d76f93c5cd4f5280c8024933e5b488ef52196fd1ffe3d16fe4b8a0
Oct 30 21:45:01.260059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.260025    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"]
Oct 30 21:45:01.276809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.276774    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"]
Oct 30 21:45:01.288259 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.286958    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"]
Oct 30 21:45:01.947284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.947243    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" event={"ID":"deefa074-2e46-4f8f-b6e2-6d9b219e3d13","Type":"ContainerStarted","Data":"4efbdda293ee19fbc67e492f184b707147a5a3128069631f7bb3db7ad3948015"}
Oct 30 21:45:01.947788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.947309    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" event={"ID":"deefa074-2e46-4f8f-b6e2-6d9b219e3d13","Type":"ContainerStarted","Data":"4ecf5a6038031beeae1e4725f1476926c22d7dc5dc444649b1790fc21286bae8"}
Oct 30 21:45:01.949129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.949102    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" event={"ID":"c4815255-f0a6-41c8-801b-17f2b1ff88f1","Type":"ContainerStarted","Data":"4aa47f882d23282663178218ba176cf7a320bcc1a48f23aaff08a2cd9ec7e100"}
Oct 30 21:45:01.949267 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.949147    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" event={"ID":"c4815255-f0a6-41c8-801b-17f2b1ff88f1","Type":"ContainerStarted","Data":"2f41ead709d76f93c5cd4f5280c8024933e5b488ef52196fd1ffe3d16fe4b8a0"}
Oct 30 21:45:01.950988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.950962    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" event={"ID":"0495337a-833d-49a6-ab96-e0a93995d335","Type":"ContainerStarted","Data":"9ebc0b86c460f29a0ae037f726fd69b218b911a3c76e098dd8793ae358f1b243"}
Oct 30 21:45:01.951101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.951003    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" event={"ID":"0495337a-833d-49a6-ab96-e0a93995d335","Type":"ContainerStarted","Data":"13a3c979dcd931b81e817876ab2596c4867e252a862121f949908611c5e33c47"}
Oct 30 21:45:01.952587 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.952561    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" event={"ID":"eb72426b-4170-4589-af8d-4f117cb92f2c","Type":"ContainerStarted","Data":"71f85f0f11009d1de3d2283e82dca73ee825a5b531fa42a1e0e4c0ca57d15619"}
Oct 30 21:45:01.952731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.952604    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" event={"ID":"eb72426b-4170-4589-af8d-4f117cb92f2c","Type":"ContainerStarted","Data":"f03d616aac9759e4655a041228e87b316e6a7f88af21ac8e6075404785198300"}
Oct 30 21:45:01.966881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.966827    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" podStartSLOduration=1.966813841 podStartE2EDuration="1.966813841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.964595135 +0000 UTC m=+2610.314322531" watchObservedRunningTime="2026-10-30 21:45:01.966813841 +0000 UTC m=+2610.316541237"
Oct 30 21:45:01.985307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:01.985254    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" podStartSLOduration=1.985240617 podStartE2EDuration="1.985240617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.981234874 +0000 UTC m=+2610.330962271" watchObservedRunningTime="2026-10-30 21:45:01.985240617 +0000 UTC m=+2610.334968012"
Oct 30 21:45:02.001888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.001838    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" podStartSLOduration=2.001822833 podStartE2EDuration="2.001822833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:01.995892227 +0000 UTC m=+2610.345619620" watchObservedRunningTime="2026-10-30 21:45:02.001822833 +0000 UTC m=+2610.351550274"
Oct 30 21:45:02.013347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.013288    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" podStartSLOduration=2.013270045 podStartE2EDuration="2.013270045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.011520388 +0000 UTC m=+2610.361247821" watchObservedRunningTime="2026-10-30 21:45:02.013270045 +0000 UTC m=+2610.362997441"
Oct 30 21:45:02.964183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.964150    4633 generic.go:356] "Generic (PLEG): container finished" podID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerID="4aa47f882d23282663178218ba176cf7a320bcc1a48f23aaff08a2cd9ec7e100" exitCode=60
Oct 30 21:45:02.964776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.964237    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" event={"ID":"c4815255-f0a6-41c8-801b-17f2b1ff88f1","Type":"ContainerDied","Data":"4aa47f882d23282663178218ba176cf7a320bcc1a48f23aaff08a2cd9ec7e100"}
Oct 30 21:45:02.964776 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.964568    4633 scope.go:122] "RemoveContainer" containerID="4aa47f882d23282663178218ba176cf7a320bcc1a48f23aaff08a2cd9ec7e100"
Oct 30 21:45:02.966217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.966196    4633 generic.go:356] "Generic (PLEG): container finished" podID="0495337a-833d-49a6-ab96-e0a93995d335" containerID="9ebc0b86c460f29a0ae037f726fd69b218b911a3c76e098dd8793ae358f1b243" exitCode=60
Oct 30 21:45:02.966346 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.966289    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" event={"ID":"0495337a-833d-49a6-ab96-e0a93995d335","Type":"ContainerDied","Data":"9ebc0b86c460f29a0ae037f726fd69b218b911a3c76e098dd8793ae358f1b243"}
Oct 30 21:45:02.966602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.966532    4633 scope.go:122] "RemoveContainer" containerID="9ebc0b86c460f29a0ae037f726fd69b218b911a3c76e098dd8793ae358f1b243"
Oct 30 21:45:02.968419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.968399    4633 generic.go:356] "Generic (PLEG): container finished" podID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerID="71f85f0f11009d1de3d2283e82dca73ee825a5b531fa42a1e0e4c0ca57d15619" exitCode=60
Oct 30 21:45:02.968522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.968469    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" event={"ID":"eb72426b-4170-4589-af8d-4f117cb92f2c","Type":"ContainerDied","Data":"71f85f0f11009d1de3d2283e82dca73ee825a5b531fa42a1e0e4c0ca57d15619"}
Oct 30 21:45:02.968823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.968801    4633 scope.go:122] "RemoveContainer" containerID="71f85f0f11009d1de3d2283e82dca73ee825a5b531fa42a1e0e4c0ca57d15619"
Oct 30 21:45:02.970965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.970946    4633 generic.go:356] "Generic (PLEG): container finished" podID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerID="4efbdda293ee19fbc67e492f184b707147a5a3128069631f7bb3db7ad3948015" exitCode=60
Oct 30 21:45:02.971091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.971032    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" event={"ID":"deefa074-2e46-4f8f-b6e2-6d9b219e3d13","Type":"ContainerDied","Data":"4efbdda293ee19fbc67e492f184b707147a5a3128069631f7bb3db7ad3948015"}
Oct 30 21:45:02.971244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:02.971225    4633 scope.go:122] "RemoveContainer" containerID="4efbdda293ee19fbc67e492f184b707147a5a3128069631f7bb3db7ad3948015"
Oct 30 21:45:03.646324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:03.644404    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:45:05.101619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.101534    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"]
Oct 30 21:45:05.214945 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.214908    4633 generic.go:356] "Generic (PLEG): container finished" podID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerID="b315e5e6ab5c825aff20dd34ea41f2003991585c667380d7f20b3b64d469f7d5" exitCode=60
Oct 30 21:45:05.215126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.215000    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" event={"ID":"c4815255-f0a6-41c8-801b-17f2b1ff88f1","Type":"ContainerDied","Data":"b315e5e6ab5c825aff20dd34ea41f2003991585c667380d7f20b3b64d469f7d5"}
Oct 30 21:45:05.215126 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.215099    4633 scope.go:122] "RemoveContainer" containerID="4aa47f882d23282663178218ba176cf7a320bcc1a48f23aaff08a2cd9ec7e100"
Oct 30 21:45:05.215452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.215436    4633 scope.go:122] "RemoveContainer" containerID="b315e5e6ab5c825aff20dd34ea41f2003991585c667380d7f20b3b64d469f7d5"
Oct 30 21:45:05.215969 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:05.215943    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj_odh-ai-gateway-infra(c4815255-f0a6-41c8-801b-17f2b1ff88f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1"
Oct 30 21:45:05.217458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.217435    4633 generic.go:356] "Generic (PLEG): container finished" podID="0495337a-833d-49a6-ab96-e0a93995d335" containerID="e7330a54cf966b5bb91562ae2287f2d768aaaa20d0c7ca89e5b6463baaed3a69" exitCode=60
Oct 30 21:45:05.217563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.217465    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" event={"ID":"0495337a-833d-49a6-ab96-e0a93995d335","Type":"ContainerDied","Data":"e7330a54cf966b5bb91562ae2287f2d768aaaa20d0c7ca89e5b6463baaed3a69"}
Oct 30 21:45:05.217912 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.217894    4633 scope.go:122] "RemoveContainer" containerID="e7330a54cf966b5bb91562ae2287f2d768aaaa20d0c7ca89e5b6463baaed3a69"
Oct 30 21:45:05.218287 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:05.218264    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-twf4t_odh-ai-gateway-infra(0495337a-833d-49a6-ab96-e0a93995d335)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" podUID="0495337a-833d-49a6-ab96-e0a93995d335"
Oct 30 21:45:05.219906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.219885    4633 generic.go:356] "Generic (PLEG): container finished" podID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerID="ff2253b0b4bfbc2dc77a072f9acd3702161dccbe2b210bb6a527e530e35ee33f" exitCode=60
Oct 30 21:45:05.220009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.219954    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" event={"ID":"eb72426b-4170-4589-af8d-4f117cb92f2c","Type":"ContainerDied","Data":"ff2253b0b4bfbc2dc77a072f9acd3702161dccbe2b210bb6a527e530e35ee33f"}
Oct 30 21:45:05.220218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.220201    4633 scope.go:122] "RemoveContainer" containerID="ff2253b0b4bfbc2dc77a072f9acd3702161dccbe2b210bb6a527e530e35ee33f"
Oct 30 21:45:05.220568 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:05.220544    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66_odh-ai-gateway-infra(eb72426b-4170-4589-af8d-4f117cb92f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c"
Oct 30 21:45:05.222274 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.222253    4633 generic.go:356] "Generic (PLEG): container finished" podID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerID="fbfa311b9d04f85f898048d276fe44cc0d3b9546f74c90fcf80ea621fb363877" exitCode=60
Oct 30 21:45:05.222372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.222359    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" event={"ID":"deefa074-2e46-4f8f-b6e2-6d9b219e3d13","Type":"ContainerDied","Data":"fbfa311b9d04f85f898048d276fe44cc0d3b9546f74c90fcf80ea621fb363877"}
Oct 30 21:45:05.222613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.222596    4633 scope.go:122] "RemoveContainer" containerID="fbfa311b9d04f85f898048d276fe44cc0d3b9546f74c90fcf80ea621fb363877"
Oct 30 21:45:05.223009 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:05.222983    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9_odh-ai-gateway-infra(deefa074-2e46-4f8f-b6e2-6d9b219e3d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13"
Oct 30 21:45:05.230391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.230365    4633 scope.go:122] "RemoveContainer" containerID="9ebc0b86c460f29a0ae037f726fd69b218b911a3c76e098dd8793ae358f1b243"
Oct 30 21:45:05.245330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.245306    4633 scope.go:122] "RemoveContainer" containerID="71f85f0f11009d1de3d2283e82dca73ee825a5b531fa42a1e0e4c0ca57d15619"
Oct 30 21:45:05.262149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.262124    4633 scope.go:122] "RemoveContainer" containerID="4efbdda293ee19fbc67e492f184b707147a5a3128069631f7bb3db7ad3948015"
Oct 30 21:45:05.645807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.642652    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:45:05.740340 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:05.740308    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1ed97/agc-cred-bb672c18dcea: secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:05.740340 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:05.740336    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k: secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:05.740710 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:05.740396    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials podName:972f8efc-fe31-4e9f-8f39-09725ca385fa nodeName:}" failed. No retries permitted until 2026-10-30 21:45:06.2403749 +0000 UTC m=+2614.590102292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" (UID: "972f8efc-fe31-4e9f-8f39-09725ca385fa") : secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:05.811841 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:05.811812    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"]
Oct 30 21:45:06.284984 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:06.282638    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1ed97/agc-cred-bb672c18dcea: secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:06.284984 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:06.282697    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k: secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:06.284984 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:06.282772    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials podName:972f8efc-fe31-4e9f-8f39-09725ca385fa nodeName:}" failed. No retries permitted until 2026-10-30 21:45:07.282751722 +0000 UTC m=+2615.632479119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" (UID: "972f8efc-fe31-4e9f-8f39-09725ca385fa") : secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:06.430696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:06.420181    4633 scope.go:122] "RemoveContainer" containerID="e7330a54cf966b5bb91562ae2287f2d768aaaa20d0c7ca89e5b6463baaed3a69"
Oct 30 21:45:06.436217 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:06.436175    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-twf4t_odh-ai-gateway-infra(0495337a-833d-49a6-ab96-e0a93995d335)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" podUID="0495337a-833d-49a6-ab96-e0a93995d335"
Oct 30 21:45:06.441116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:06.441092    4633 scope.go:122] "RemoveContainer" containerID="ff2253b0b4bfbc2dc77a072f9acd3702161dccbe2b210bb6a527e530e35ee33f"
Oct 30 21:45:06.441981 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:06.441950    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66_odh-ai-gateway-infra(eb72426b-4170-4589-af8d-4f117cb92f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c"
Oct 30 21:45:06.444904 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:06.444880    4633 scope.go:122] "RemoveContainer" containerID="fbfa311b9d04f85f898048d276fe44cc0d3b9546f74c90fcf80ea621fb363877"
Oct 30 21:45:06.446832 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:06.446803    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9_odh-ai-gateway-infra(deefa074-2e46-4f8f-b6e2-6d9b219e3d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13"
Oct 30 21:45:06.451209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:06.451188    4633 scope.go:122] "RemoveContainer" containerID="b315e5e6ab5c825aff20dd34ea41f2003991585c667380d7f20b3b64d469f7d5"
Oct 30 21:45:06.451628 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:06.451602    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj_odh-ai-gateway-infra(c4815255-f0a6-41c8-801b-17f2b1ff88f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1"
Oct 30 21:45:07.298334 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:07.298298    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1ed97/agc-cred-bb672c18dcea: secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:07.298334 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:07.298329    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k: secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:07.298810 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:07.298383    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials podName:972f8efc-fe31-4e9f-8f39-09725ca385fa nodeName:}" failed. No retries permitted until 2026-10-30 21:45:09.298368598 +0000 UTC m=+2617.648095989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" (UID: "972f8efc-fe31-4e9f-8f39-09725ca385fa") : secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:07.961722 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:07.961253    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:45:09.369578 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:09.369549    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-f1ed97/agc-cred-bb672c18dcea: secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:09.369578 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:09.369574    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k: secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:09.370144 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:09.369624    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials podName:972f8efc-fe31-4e9f-8f39-09725ca385fa nodeName:}" failed. No retries permitted until 2026-10-30 21:45:13.369610528 +0000 UTC m=+2621.719337903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" (UID: "972f8efc-fe31-4e9f-8f39-09725ca385fa") : secret "agc-cred-bb672c18dcea" not found
Oct 30 21:45:10.102244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.102189    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk" podUID="b6ec9a01-f868-4b7f-ba04-40a340ad26b2" containerName="payload-processing" containerID="cri-o://6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1" gracePeriod=25
Oct 30 21:45:10.394396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.394358    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:45:10.488559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.488521    4633 generic.go:356] "Generic (PLEG): container finished" podID="b6ec9a01-f868-4b7f-ba04-40a340ad26b2" containerID="6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1" exitCode=0
Oct 30 21:45:10.488791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.488568    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"
Oct 30 21:45:10.488791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.488611    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk" event={"ID":"b6ec9a01-f868-4b7f-ba04-40a340ad26b2","Type":"ContainerDied","Data":"6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1"}
Oct 30 21:45:10.488791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.488681    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk" event={"ID":"b6ec9a01-f868-4b7f-ba04-40a340ad26b2","Type":"ContainerDied","Data":"0333b84401f389f37964800d46ac72ee10cdf70313b0efad9288461962b6cc05"}
Oct 30 21:45:10.488791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.488711    4633 scope.go:122] "RemoveContainer" containerID="6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1"
Oct 30 21:45:10.496825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.496800    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-routing-overlay-volume\") pod \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") "
Oct 30 21:45:10.496961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.496950    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-kube-api-access-w4dpt\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-kube-api-access-w4dpt\") pod \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") "
Oct 30 21:45:10.497022 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.496973    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-provider-credentials\") pod \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") "
Oct 30 21:45:10.497061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.497038    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-plugins-config-volume\") pod \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\" (UID: \"b6ec9a01-f868-4b7f-ba04-40a340ad26b2\") "
Oct 30 21:45:10.497257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.497232    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-routing-overlay-volume" pod "b6ec9a01-f868-4b7f-ba04-40a340ad26b2" (UID: "b6ec9a01-f868-4b7f-ba04-40a340ad26b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:10.497634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.497558    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-plugins-config-volume" pod "b6ec9a01-f868-4b7f-ba04-40a340ad26b2" (UID: "b6ec9a01-f868-4b7f-ba04-40a340ad26b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:10.498049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.498025    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:10.498137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.498056    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:10.499466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.499438    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-provider-credentials" pod "b6ec9a01-f868-4b7f-ba04-40a340ad26b2" (UID: "b6ec9a01-f868-4b7f-ba04-40a340ad26b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:10.499466 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.499456    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-kube-api-access-w4dpt" pod "b6ec9a01-f868-4b7f-ba04-40a340ad26b2" (UID: "b6ec9a01-f868-4b7f-ba04-40a340ad26b2"). InnerVolumeSpecName "kube-api-access-w4dpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:10.501565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.501544    4633 scope.go:122] "RemoveContainer" containerID="6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1"
Oct 30 21:45:10.501934 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:10.501900    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1\": container with ID starting with 6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1 not found: ID does not exist" containerID="6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1"
Oct 30 21:45:10.502005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.501944    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1"} err="failed to get container status \"6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1\": rpc error: code = NotFound desc = could not find container \"6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1\": container with ID starting with 6e7aacff71bf508ced396da7b11be195e90544779cd53475e2fc60eb55e4d2e1 not found: ID does not exist"
Oct 30 21:45:10.611651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.605295    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dpt\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-kube-api-access-w4dpt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:10.611651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.605338    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6ec9a01-f868-4b7f-ba04-40a340ad26b2-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:10.815879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.815852    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"]
Oct 30 21:45:10.820797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:10.820770    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-b7c5t4ptk"]
Oct 30 21:45:11.459724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.452201    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" podUID="972f8efc-fe31-4e9f-8f39-09725ca385fa" containerName="payload-processing" containerID="cri-o://ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587" gracePeriod=25
Oct 30 21:45:11.761001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.760975    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:45:11.793598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.793551    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ec9a01-f868-4b7f-ba04-40a340ad26b2" path="/var/lib/kubelet/pods/b6ec9a01-f868-4b7f-ba04-40a340ad26b2/volumes"
Oct 30 21:45:11.827040 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.827012    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-routing-overlay-volume\") pod \"972f8efc-fe31-4e9f-8f39-09725ca385fa\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") "
Oct 30 21:45:11.827326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.827142    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-kube-api-access-96b8z\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-kube-api-access-96b8z\") pod \"972f8efc-fe31-4e9f-8f39-09725ca385fa\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") "
Oct 30 21:45:11.827326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.827297    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-plugins-config-volume\") pod \"972f8efc-fe31-4e9f-8f39-09725ca385fa\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") "
Oct 30 21:45:11.827464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.827391    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-routing-overlay-volume" pod "972f8efc-fe31-4e9f-8f39-09725ca385fa" (UID: "972f8efc-fe31-4e9f-8f39-09725ca385fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:11.827522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.827499    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials\") pod \"972f8efc-fe31-4e9f-8f39-09725ca385fa\" (UID: \"972f8efc-fe31-4e9f-8f39-09725ca385fa\") "
Oct 30 21:45:11.827758 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.827735    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-plugins-config-volume" pod "972f8efc-fe31-4e9f-8f39-09725ca385fa" (UID: "972f8efc-fe31-4e9f-8f39-09725ca385fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:11.828179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.828159    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:11.828260 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.828184    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/972f8efc-fe31-4e9f-8f39-09725ca385fa-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:11.829763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.829731    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-kube-api-access-96b8z" pod "972f8efc-fe31-4e9f-8f39-09725ca385fa" (UID: "972f8efc-fe31-4e9f-8f39-09725ca385fa"). InnerVolumeSpecName "kube-api-access-96b8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:11.830101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.830076    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials" pod "972f8efc-fe31-4e9f-8f39-09725ca385fa" (UID: "972f8efc-fe31-4e9f-8f39-09725ca385fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:11.929613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.929573    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:11.929613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:11.929605    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96b8z\" (UniqueName: \"kubernetes.io/projected/972f8efc-fe31-4e9f-8f39-09725ca385fa-kube-api-access-96b8z\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:12.509737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.509657    4633 generic.go:356] "Generic (PLEG): container finished" podID="972f8efc-fe31-4e9f-8f39-09725ca385fa" containerID="ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587" exitCode=0
Oct 30 21:45:12.510237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.509757    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"
Oct 30 21:45:12.510237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.509755    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" event={"ID":"972f8efc-fe31-4e9f-8f39-09725ca385fa","Type":"ContainerDied","Data":"ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587"}
Oct 30 21:45:12.510237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.509913    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k" event={"ID":"972f8efc-fe31-4e9f-8f39-09725ca385fa","Type":"ContainerDied","Data":"14f66d162ee13d002ea3ad86e490aa905a915dda4c95e908b6744959e076f625"}
Oct 30 21:45:12.510237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.509951    4633 scope.go:122] "RemoveContainer" containerID="ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587"
Oct 30 21:45:12.523467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.523450    4633 scope.go:122] "RemoveContainer" containerID="ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587"
Oct 30 21:45:12.523794 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:12.523774    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587\": container with ID starting with ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587 not found: ID does not exist" containerID="ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587"
Oct 30 21:45:12.523849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.523802    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587"} err="failed to get container status \"ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587\": rpc error: code = NotFound desc = could not find container \"ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587\": container with ID starting with ebbc36d5e5f5daef57f4c8ee1c1018af7a32dab994dba526f5430a5c97b7c587 not found: ID does not exist"
Oct 30 21:45:12.543177 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.543155    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"]
Oct 30 21:45:12.548846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:12.548822    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-55462rj2k"]
Oct 30 21:45:13.793907 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:13.793865    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972f8efc-fe31-4e9f-8f39-09725ca385fa" path="/var/lib/kubelet/pods/972f8efc-fe31-4e9f-8f39-09725ca385fa/volumes"
Oct 30 21:45:15.450197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.450149    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"]
Oct 30 21:45:15.453569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.452205    4633 cpu_manager.go:414] "containerMap: removing container" podUID="972f8efc-fe31-4e9f-8f39-09725ca385fa" containerName="payload-processing"
Oct 30 21:45:15.453825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.453806    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972f8efc-fe31-4e9f-8f39-09725ca385fa" containerName="payload-processing"
Oct 30 21:45:15.453990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.453977    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b6ec9a01-f868-4b7f-ba04-40a340ad26b2" containerName="payload-processing"
Oct 30 21:45:15.454072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.454062    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ec9a01-f868-4b7f-ba04-40a340ad26b2" containerName="payload-processing"
Oct 30 21:45:15.454447 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.454431    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="972f8efc-fe31-4e9f-8f39-09725ca385fa" containerName="payload-processing"
Oct 30 21:45:15.454562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.454552    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ec9a01-f868-4b7f-ba04-40a340ad26b2" containerName="payload-processing"
Oct 30 21:45:15.462659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.462631    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"]
Oct 30 21:45:15.462996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.462806    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.469654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.469627    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8431de-openshift-default-dockercfg-w6wgl\""
Oct 30 21:45:15.786839 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.786723    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-certs\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.787187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.787158    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.787410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.787379    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-podinfo\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.787774 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.787589    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-credential-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.788251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.788229    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-token\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.788409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.788378    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxkz\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-kube-api-access-6xxkz\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.788637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.788618    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-envoy\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.789023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.789004    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-data\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.791305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.791275    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4294be0-dd4b-4442-8f5e-9644df33121c-istiod-ca-cert\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.831796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.831608    4633 scope.go:122] "RemoveContainer" containerID="b315e5e6ab5c825aff20dd34ea41f2003991585c667380d7f20b3b64d469f7d5"
Oct 30 21:45:15.832916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.832891    4633 scope.go:122] "RemoveContainer" containerID="fbfa311b9d04f85f898048d276fe44cc0d3b9546f74c90fcf80ea621fb363877"
Oct 30 21:45:15.892867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.892817    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-credential-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.893041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.892916    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-token\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.893041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.892964    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxkz\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-kube-api-access-6xxkz\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.893041 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.893016    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-envoy\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.893188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.893170    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-data\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.893236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.893201    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4294be0-dd4b-4442-8f5e-9644df33121c-istiod-ca-cert\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.893391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.893371    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-certs\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.893443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.893428    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.893503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.893490    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-podinfo\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.894130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.894101    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-certs\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.894130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.894123    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.894279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.894187    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-data\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.894279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.894205    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4294be0-dd4b-4442-8f5e-9644df33121c-istiod-ca-cert\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.894550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.894524    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-credential-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.896162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.896142    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-envoy\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.896557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.896540    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-podinfo\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.903995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.903973    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-token\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.904129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.904043    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxkz\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-kube-api-access-6xxkz\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:15.935933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:15.935900    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:16.239049 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:45:16.239015    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4294be0_dd4b_4442_8f5e_9644df33121c.slice/crio-325d962841a1b502e1bad7b68c5a2f4236dcaa29adc9a975f3a5caae4e686585 WatchSource:0}: Error finding container 325d962841a1b502e1bad7b68c5a2f4236dcaa29adc9a975f3a5caae4e686585: Status 404 returned error can't find the container with id 325d962841a1b502e1bad7b68c5a2f4236dcaa29adc9a975f3a5caae4e686585
Oct 30 21:45:16.240079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.240055    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"]
Oct 30 21:45:16.242362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.242323    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:16.242480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.242402    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:16.242480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.242466    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:16.783800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.783696    4633 scope.go:122] "RemoveContainer" containerID="e7330a54cf966b5bb91562ae2287f2d768aaaa20d0c7ca89e5b6463baaed3a69"
Oct 30 21:45:16.794732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.794697    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" event={"ID":"c4815255-f0a6-41c8-801b-17f2b1ff88f1","Type":"ContainerStarted","Data":"a3aea0b0d802e9e39fdd42015345314a9b8c6565657b70ef01047987f2ba1f85"}
Oct 30 21:45:16.797453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.797423    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" event={"ID":"deefa074-2e46-4f8f-b6e2-6d9b219e3d13","Type":"ContainerStarted","Data":"5c3fa518036a0e9dc0dabe60cd3a0f96aff3acd15984eb156d14aa731dad5404"}
Oct 30 21:45:16.799543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.799517    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" event={"ID":"a4294be0-dd4b-4442-8f5e-9644df33121c","Type":"ContainerStarted","Data":"07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385"}
Oct 30 21:45:16.799659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.799566    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" event={"ID":"a4294be0-dd4b-4442-8f5e-9644df33121c","Type":"ContainerStarted","Data":"325d962841a1b502e1bad7b68c5a2f4236dcaa29adc9a975f3a5caae4e686585"}
Oct 30 21:45:16.866130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.866068    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podStartSLOduration=1.86605285 podStartE2EDuration="1.86605285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:16.86256753 +0000 UTC m=+2625.212294930" watchObservedRunningTime="2026-10-30 21:45:16.86605285 +0000 UTC m=+2625.215780246"
Oct 30 21:45:16.936581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:16.936547    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:17.793575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.793547    4633 scope.go:122] "RemoveContainer" containerID="ff2253b0b4bfbc2dc77a072f9acd3702161dccbe2b210bb6a527e530e35ee33f"
Oct 30 21:45:17.810305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.810280    4633 generic.go:356] "Generic (PLEG): container finished" podID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerID="a3aea0b0d802e9e39fdd42015345314a9b8c6565657b70ef01047987f2ba1f85" exitCode=60
Oct 30 21:45:17.810410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.810358    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" event={"ID":"c4815255-f0a6-41c8-801b-17f2b1ff88f1","Type":"ContainerDied","Data":"a3aea0b0d802e9e39fdd42015345314a9b8c6565657b70ef01047987f2ba1f85"}
Oct 30 21:45:17.810494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.810426    4633 scope.go:122] "RemoveContainer" containerID="b315e5e6ab5c825aff20dd34ea41f2003991585c667380d7f20b3b64d469f7d5"
Oct 30 21:45:17.810854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.810835    4633 scope.go:122] "RemoveContainer" containerID="a3aea0b0d802e9e39fdd42015345314a9b8c6565657b70ef01047987f2ba1f85"
Oct 30 21:45:17.811296 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:17.811267    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj_odh-ai-gateway-infra(c4815255-f0a6-41c8-801b-17f2b1ff88f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1"
Oct 30 21:45:17.812456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.812428    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" event={"ID":"0495337a-833d-49a6-ab96-e0a93995d335","Type":"ContainerStarted","Data":"30f2436e1d2b74e4e3c854f9bd8bc09e8353fad37b32f4176bbca847c977a13d"}
Oct 30 21:45:17.814603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.814585    4633 generic.go:356] "Generic (PLEG): container finished" podID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerID="5c3fa518036a0e9dc0dabe60cd3a0f96aff3acd15984eb156d14aa731dad5404" exitCode=60
Oct 30 21:45:17.814683 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.814635    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" event={"ID":"deefa074-2e46-4f8f-b6e2-6d9b219e3d13","Type":"ContainerDied","Data":"5c3fa518036a0e9dc0dabe60cd3a0f96aff3acd15984eb156d14aa731dad5404"}
Oct 30 21:45:17.815075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.815044    4633 scope.go:122] "RemoveContainer" containerID="5c3fa518036a0e9dc0dabe60cd3a0f96aff3acd15984eb156d14aa731dad5404"
Oct 30 21:45:17.815423 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:17.815397    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9_odh-ai-gateway-infra(deefa074-2e46-4f8f-b6e2-6d9b219e3d13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13"
Oct 30 21:45:17.842800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.842780    4633 scope.go:122] "RemoveContainer" containerID="fbfa311b9d04f85f898048d276fe44cc0d3b9546f74c90fcf80ea621fb363877"
Oct 30 21:45:17.845094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.845073    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"]
Oct 30 21:45:17.867707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.867675    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"]
Oct 30 21:45:17.943619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:17.943590    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:18.827775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.827748    4633 generic.go:356] "Generic (PLEG): container finished" podID="0495337a-833d-49a6-ab96-e0a93995d335" containerID="30f2436e1d2b74e4e3c854f9bd8bc09e8353fad37b32f4176bbca847c977a13d" exitCode=60
Oct 30 21:45:18.828170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.827823    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" event={"ID":"0495337a-833d-49a6-ab96-e0a93995d335","Type":"ContainerDied","Data":"30f2436e1d2b74e4e3c854f9bd8bc09e8353fad37b32f4176bbca847c977a13d"}
Oct 30 21:45:18.828170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.827896    4633 scope.go:122] "RemoveContainer" containerID="e7330a54cf966b5bb91562ae2287f2d768aaaa20d0c7ca89e5b6463baaed3a69"
Oct 30 21:45:18.828403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.828385    4633 scope.go:122] "RemoveContainer" containerID="30f2436e1d2b74e4e3c854f9bd8bc09e8353fad37b32f4176bbca847c977a13d"
Oct 30 21:45:18.828916 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:18.828887    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-twf4t_odh-ai-gateway-infra(0495337a-833d-49a6-ab96-e0a93995d335)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" podUID="0495337a-833d-49a6-ab96-e0a93995d335"
Oct 30 21:45:18.830859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.830823    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" event={"ID":"eb72426b-4170-4589-af8d-4f117cb92f2c","Type":"ContainerStarted","Data":"ca26b0b403469ecd92866d4f002653323707bd05e00a7e7eba01b449bcc01191"}
Oct 30 21:45:18.831240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.831225    4633 scope.go:122] "RemoveContainer" containerID="ca26b0b403469ecd92866d4f002653323707bd05e00a7e7eba01b449bcc01191"
Oct 30 21:45:18.831660 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:18.831636    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66_odh-ai-gateway-infra(eb72426b-4170-4589-af8d-4f117cb92f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c"
Oct 30 21:45:18.833181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.833130    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:18.834326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.834305    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:45:18.942014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:18.941965    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"]
Oct 30 21:45:19.084467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.084401    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:19.139080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.139042    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:19.153209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.153186    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-kube-api-access-kg2tw\" (UniqueName: \"kubernetes.io/projected/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-kube-api-access-kg2tw\") pod \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\" (UID: \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\") "
Oct 30 21:45:19.153396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.153245    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4815255-f0a6-41c8-801b-17f2b1ff88f1-kube-api-access-hd8fr\" (UniqueName: \"kubernetes.io/projected/c4815255-f0a6-41c8-801b-17f2b1ff88f1-kube-api-access-hd8fr\") pod \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\" (UID: \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\") "
Oct 30 21:45:19.153396 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.153346    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-maas-api-service-ca\") pod \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\" (UID: \"deefa074-2e46-4f8f-b6e2-6d9b219e3d13\") "
Oct 30 21:45:19.153537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.153451    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4815255-f0a6-41c8-801b-17f2b1ff88f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4815255-f0a6-41c8-801b-17f2b1ff88f1-maas-api-service-ca\") pod \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\" (UID: \"c4815255-f0a6-41c8-801b-17f2b1ff88f1\") "
Oct 30 21:45:19.153892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.153820    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-maas-api-service-ca" pod "deefa074-2e46-4f8f-b6e2-6d9b219e3d13" (UID: "deefa074-2e46-4f8f-b6e2-6d9b219e3d13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:19.154156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.154132    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:19.154272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.154156    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4815255-f0a6-41c8-801b-17f2b1ff88f1-maas-api-service-ca" pod "c4815255-f0a6-41c8-801b-17f2b1ff88f1" (UID: "c4815255-f0a6-41c8-801b-17f2b1ff88f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:19.156882 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.156803    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4815255-f0a6-41c8-801b-17f2b1ff88f1-kube-api-access-hd8fr" pod "c4815255-f0a6-41c8-801b-17f2b1ff88f1" (UID: "c4815255-f0a6-41c8-801b-17f2b1ff88f1"). InnerVolumeSpecName "kube-api-access-hd8fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:19.157328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.157305    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-kube-api-access-kg2tw" pod "deefa074-2e46-4f8f-b6e2-6d9b219e3d13" (UID: "deefa074-2e46-4f8f-b6e2-6d9b219e3d13"). InnerVolumeSpecName "kube-api-access-kg2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:19.311692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.311224    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd8fr\" (UniqueName: \"kubernetes.io/projected/c4815255-f0a6-41c8-801b-17f2b1ff88f1-kube-api-access-hd8fr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:19.311692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.311263    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4815255-f0a6-41c8-801b-17f2b1ff88f1-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:19.311692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.311278    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg2tw\" (UniqueName: \"kubernetes.io/projected/deefa074-2e46-4f8f-b6e2-6d9b219e3d13-kube-api-access-kg2tw\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:19.794324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.794292    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"]
Oct 30 21:45:19.845272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.845232    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9" event={"ID":"deefa074-2e46-4f8f-b6e2-6d9b219e3d13","Type":"ContainerDied","Data":"4ecf5a6038031beeae1e4725f1476926c22d7dc5dc444649b1790fc21286bae8"}
Oct 30 21:45:19.845627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.845310    4633 scope.go:122] "RemoveContainer" containerID="5c3fa518036a0e9dc0dabe60cd3a0f96aff3acd15984eb156d14aa731dad5404"
Oct 30 21:45:19.845627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.845406    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"
Oct 30 21:45:19.848251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.848229    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj" event={"ID":"c4815255-f0a6-41c8-801b-17f2b1ff88f1","Type":"ContainerDied","Data":"2f41ead709d76f93c5cd4f5280c8024933e5b488ef52196fd1ffe3d16fe4b8a0"}
Oct 30 21:45:19.848455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.848327    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"
Oct 30 21:45:19.858172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.858139    4633 generic.go:356] "Generic (PLEG): container finished" podID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerID="ca26b0b403469ecd92866d4f002653323707bd05e00a7e7eba01b449bcc01191" exitCode=60
Oct 30 21:45:19.858386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.858362    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" event={"ID":"eb72426b-4170-4589-af8d-4f117cb92f2c","Type":"ContainerDied","Data":"ca26b0b403469ecd92866d4f002653323707bd05e00a7e7eba01b449bcc01191"}
Oct 30 21:45:19.859002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.858981    4633 scope.go:122] "RemoveContainer" containerID="ca26b0b403469ecd92866d4f002653323707bd05e00a7e7eba01b449bcc01191"
Oct 30 21:45:19.859516 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:19.859492    4633 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66_odh-ai-gateway-infra(eb72426b-4170-4589-af8d-4f117cb92f2c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c"
Oct 30 21:45:19.861226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.861196    4633 scope.go:122] "RemoveContainer" containerID="a3aea0b0d802e9e39fdd42015345314a9b8c6565657b70ef01047987f2ba1f85"
Oct 30 21:45:19.884951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.884930    4633 scope.go:122] "RemoveContainer" containerID="ff2253b0b4bfbc2dc77a072f9acd3702161dccbe2b210bb6a527e530e35ee33f"
Oct 30 21:45:19.916973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.916933    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"]
Oct 30 21:45:19.970089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:19.964081    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889945-6tpvj"]
Oct 30 21:45:20.008884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.008795    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"]
Oct 30 21:45:20.129051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.129022    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:20.139600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.139565    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"]
Oct 30 21:45:20.151242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.151210    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889945-2qjp9"]
Oct 30 21:45:20.267870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.267838    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0495337a-833d-49a6-ab96-e0a93995d335-kube-api-access-p5wnq\" (UniqueName: \"kubernetes.io/projected/0495337a-833d-49a6-ab96-e0a93995d335-kube-api-access-p5wnq\") pod \"0495337a-833d-49a6-ab96-e0a93995d335\" (UID: \"0495337a-833d-49a6-ab96-e0a93995d335\") "
Oct 30 21:45:20.268038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.267907    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0495337a-833d-49a6-ab96-e0a93995d335-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0495337a-833d-49a6-ab96-e0a93995d335-maas-api-service-ca\") pod \"0495337a-833d-49a6-ab96-e0a93995d335\" (UID: \"0495337a-833d-49a6-ab96-e0a93995d335\") "
Oct 30 21:45:20.268311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.268283    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0495337a-833d-49a6-ab96-e0a93995d335-maas-api-service-ca" pod "0495337a-833d-49a6-ab96-e0a93995d335" (UID: "0495337a-833d-49a6-ab96-e0a93995d335"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.268840 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.268815    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0495337a-833d-49a6-ab96-e0a93995d335-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:20.270280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.270251    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0495337a-833d-49a6-ab96-e0a93995d335-kube-api-access-p5wnq" pod "0495337a-833d-49a6-ab96-e0a93995d335" (UID: "0495337a-833d-49a6-ab96-e0a93995d335"). InnerVolumeSpecName "kube-api-access-p5wnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:20.370027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.369938    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5wnq\" (UniqueName: \"kubernetes.io/projected/0495337a-833d-49a6-ab96-e0a93995d335-kube-api-access-p5wnq\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:20.871304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.871274    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"]
Oct 30 21:45:20.871784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.871623    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"
Oct 30 21:45:20.872498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872465    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.872498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872494    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872553    4633 cpu_manager.go:414] "containerMap: removing container" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872562    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872579    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872586    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872596    4633 cpu_manager.go:414] "containerMap: removing container" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872605    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872644    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872652    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872700    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.872755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872708    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.873193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872885    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.873193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872897    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.873193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872906    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.873193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872917    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.873193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872928    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.873193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872944    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.873193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.872954    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.877812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.877786    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t" event={"ID":"0495337a-833d-49a6-ab96-e0a93995d335","Type":"ContainerDied","Data":"13a3c979dcd931b81e817876ab2596c4867e252a862121f949908611c5e33c47"}
Oct 30 21:45:20.877915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.877841    4633 scope.go:122] "RemoveContainer" containerID="30f2436e1d2b74e4e3c854f9bd8bc09e8353fad37b32f4176bbca847c977a13d"
Oct 30 21:45:20.877915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.877872    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:20.887951 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.887911    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8431de-dockercfg-f954t\""
Oct 30 21:45:20.889652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.889629    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8431de\""
Oct 30 21:45:20.894661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.894242    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"]
Oct 30 21:45:20.911362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.911142    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"]
Oct 30 21:45:20.912349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.912325    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.912349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.912346    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" containerName="cleanup"
Oct 30 21:45:20.912349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.912354    4633 cpu_manager.go:414] "containerMap: removing container" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.912603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.912359    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.912603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.912405    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.912603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.912411    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.912603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.912525    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" containerName="cleanup"
Oct 30 21:45:20.912603 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.912537    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0495337a-833d-49a6-ab96-e0a93995d335" containerName="cleanup"
Oct 30 21:45:20.916782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.916761    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:20.943469 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.943430    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"]
Oct 30 21:45:20.981129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.980526    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jwd\" (UniqueName: \"kubernetes.io/projected/e42295e3-7338-4654-ae73-c08cfa9a4875-kube-api-access-j2jwd\") pod \"payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt\" (UID: \"e42295e3-7338-4654-ae73-c08cfa9a4875\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:20.981129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.980604    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5tr8\" (UniqueName: \"kubernetes.io/projected/49fddf86-5424-4a8a-b896-ca1aafc24165-kube-api-access-p5tr8\") pod \"payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz\" (UID: \"49fddf86-5424-4a8a-b896-ca1aafc24165\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:20.981129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.980745    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fddf86-5424-4a8a-b896-ca1aafc24165-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz\" (UID: \"49fddf86-5424-4a8a-b896-ca1aafc24165\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:20.981129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:20.981109    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42295e3-7338-4654-ae73-c08cfa9a4875-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt\" (UID: \"e42295e3-7338-4654-ae73-c08cfa9a4875\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:21.101367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.101331    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5tr8\" (UniqueName: \"kubernetes.io/projected/49fddf86-5424-4a8a-b896-ca1aafc24165-kube-api-access-p5tr8\") pod \"payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz\" (UID: \"49fddf86-5424-4a8a-b896-ca1aafc24165\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:21.101719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.101696    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fddf86-5424-4a8a-b896-ca1aafc24165-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz\" (UID: \"49fddf86-5424-4a8a-b896-ca1aafc24165\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:21.173074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.173039    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42295e3-7338-4654-ae73-c08cfa9a4875-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt\" (UID: \"e42295e3-7338-4654-ae73-c08cfa9a4875\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:21.173369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.173352    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jwd\" (UniqueName: \"kubernetes.io/projected/e42295e3-7338-4654-ae73-c08cfa9a4875-kube-api-access-j2jwd\") pod \"payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt\" (UID: \"e42295e3-7338-4654-ae73-c08cfa9a4875\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:21.198697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.164580    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fddf86-5424-4a8a-b896-ca1aafc24165-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz\" (UID: \"49fddf86-5424-4a8a-b896-ca1aafc24165\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:21.199043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.174370    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42295e3-7338-4654-ae73-c08cfa9a4875-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt\" (UID: \"e42295e3-7338-4654-ae73-c08cfa9a4875\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:21.259121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.257930    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"]
Oct 30 21:45:21.326697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.306234    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5tr8\" (UniqueName: \"kubernetes.io/projected/49fddf86-5424-4a8a-b896-ca1aafc24165-kube-api-access-p5tr8\") pod \"payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz\" (UID: \"49fddf86-5424-4a8a-b896-ca1aafc24165\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:21.327257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.327172    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-twf4t"]
Oct 30 21:45:21.329104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.329077    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jwd\" (UniqueName: \"kubernetes.io/projected/e42295e3-7338-4654-ae73-c08cfa9a4875-kube-api-access-j2jwd\") pod \"payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt\" (UID: \"e42295e3-7338-4654-ae73-c08cfa9a4875\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:21.523078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.522984    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:21.545802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.545735    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:21.591379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.589821    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:21.726512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.726479    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb72426b-4170-4589-af8d-4f117cb92f2c-kube-api-access-zvxj5\" (UniqueName: \"kubernetes.io/projected/eb72426b-4170-4589-af8d-4f117cb92f2c-kube-api-access-zvxj5\") pod \"eb72426b-4170-4589-af8d-4f117cb92f2c\" (UID: \"eb72426b-4170-4589-af8d-4f117cb92f2c\") "
Oct 30 21:45:21.726931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.726595    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb72426b-4170-4589-af8d-4f117cb92f2c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72426b-4170-4589-af8d-4f117cb92f2c-maas-api-service-ca\") pod \"eb72426b-4170-4589-af8d-4f117cb92f2c\" (UID: \"eb72426b-4170-4589-af8d-4f117cb92f2c\") "
Oct 30 21:45:21.727532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.727488    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb72426b-4170-4589-af8d-4f117cb92f2c-maas-api-service-ca" pod "eb72426b-4170-4589-af8d-4f117cb92f2c" (UID: "eb72426b-4170-4589-af8d-4f117cb92f2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:21.729748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.729722    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb72426b-4170-4589-af8d-4f117cb92f2c-kube-api-access-zvxj5" pod "eb72426b-4170-4589-af8d-4f117cb92f2c" (UID: "eb72426b-4170-4589-af8d-4f117cb92f2c"). InnerVolumeSpecName "kube-api-access-zvxj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:21.798775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.798697    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0495337a-833d-49a6-ab96-e0a93995d335" path="/var/lib/kubelet/pods/0495337a-833d-49a6-ab96-e0a93995d335/volumes"
Oct 30 21:45:21.799250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.799235    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4815255-f0a6-41c8-801b-17f2b1ff88f1" path="/var/lib/kubelet/pods/c4815255-f0a6-41c8-801b-17f2b1ff88f1/volumes"
Oct 30 21:45:21.799638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.799624    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deefa074-2e46-4f8f-b6e2-6d9b219e3d13" path="/var/lib/kubelet/pods/deefa074-2e46-4f8f-b6e2-6d9b219e3d13/volumes"
Oct 30 21:45:21.828551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.828521    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxj5\" (UniqueName: \"kubernetes.io/projected/eb72426b-4170-4589-af8d-4f117cb92f2c-kube-api-access-zvxj5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:21.828551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.828547    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72426b-4170-4589-af8d-4f117cb92f2c-maas-api-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:21.868902 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:45:21.868871    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42295e3_7338_4654_ae73_c08cfa9a4875.slice/crio-019a704fb16057d37a5b9e0d470ccccdf1b0d94266eb69050db777f800965dfe WatchSource:0}: Error finding container 019a704fb16057d37a5b9e0d470ccccdf1b0d94266eb69050db777f800965dfe: Status 404 returned error can't find the container with id 019a704fb16057d37a5b9e0d470ccccdf1b0d94266eb69050db777f800965dfe
Oct 30 21:45:21.871179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.871156    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"]
Oct 30 21:45:21.889661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.889636    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"
Oct 30 21:45:21.890053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.889659    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66" event={"ID":"eb72426b-4170-4589-af8d-4f117cb92f2c","Type":"ContainerDied","Data":"f03d616aac9759e4655a041228e87b316e6a7f88af21ac8e6075404785198300"}
Oct 30 21:45:21.890159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.889772    4633 scope.go:122] "RemoveContainer" containerID="ca26b0b403469ecd92866d4f002653323707bd05e00a7e7eba01b449bcc01191"
Oct 30 21:45:21.894184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.894154    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt" event={"ID":"e42295e3-7338-4654-ae73-c08cfa9a4875","Type":"ContainerStarted","Data":"019a704fb16057d37a5b9e0d470ccccdf1b0d94266eb69050db777f800965dfe"}
Oct 30 21:45:21.902509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.902484    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"]
Oct 30 21:45:21.947060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.947035    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"]
Oct 30 21:45:21.952435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:21.952408    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889945-v8c66"]
Oct 30 21:45:22.906294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:22.906254    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz" event={"ID":"49fddf86-5424-4a8a-b896-ca1aafc24165","Type":"ContainerStarted","Data":"154bc56a006f37a6417d319b9855d96847f89d8cba7e9b486c2f30c5c0c52e2c"}
Oct 30 21:45:22.906803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:22.906344    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz" event={"ID":"49fddf86-5424-4a8a-b896-ca1aafc24165","Type":"ContainerStarted","Data":"e4a1d6d787a1a7d84bc9a0a80b86adba0fa6f0beb732d910d8aa83feb31fa9d8"}
Oct 30 21:45:22.906803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:22.906380    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:22.908197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:22.908172    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt" event={"ID":"e42295e3-7338-4654-ae73-c08cfa9a4875","Type":"ContainerStarted","Data":"7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1"}
Oct 30 21:45:22.908320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:22.908248    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:22.947710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:22.947616    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz" podStartSLOduration=2.947599836 podStartE2EDuration="2.947599836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:22.946332295 +0000 UTC m=+2631.296059713" watchObservedRunningTime="2026-10-30 21:45:22.947599836 +0000 UTC m=+2631.297327236"
Oct 30 21:45:22.986787 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:22.986726    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt" podStartSLOduration=2.986708794 podStartE2EDuration="2.986708794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:22.979981645 +0000 UTC m=+2631.329709041" watchObservedRunningTime="2026-10-30 21:45:22.986708794 +0000 UTC m=+2631.336436192"
Oct 30 21:45:23.794504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:23.794470    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" path="/var/lib/kubelet/pods/eb72426b-4170-4589-af8d-4f117cb92f2c/volumes"
Oct 30 21:45:24.794855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:24.794809    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr" podUID="c2796e7a-8051-4b7c-80ac-14d63c153691" containerName="payload-processing" containerID="cri-o://5022ee12c2d292e061238e2fe98a0dc3065a6d739a664418018fe94dabc40876" gracePeriod=25
Oct 30 21:45:24.930794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:24.930758    4633 generic.go:356] "Generic (PLEG): container finished" podID="c2796e7a-8051-4b7c-80ac-14d63c153691" containerID="5022ee12c2d292e061238e2fe98a0dc3065a6d739a664418018fe94dabc40876" exitCode=0
Oct 30 21:45:24.930977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:24.930838    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr" event={"ID":"c2796e7a-8051-4b7c-80ac-14d63c153691","Type":"ContainerDied","Data":"5022ee12c2d292e061238e2fe98a0dc3065a6d739a664418018fe94dabc40876"}
Oct 30 21:45:25.028890 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:25.028846    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:45:25.128859 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.128832    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:45:25.280695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.279399    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-routing-overlay-volume\") pod \"c2796e7a-8051-4b7c-80ac-14d63c153691\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") "
Oct 30 21:45:25.303693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.299117    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-routing-overlay-volume" pod "c2796e7a-8051-4b7c-80ac-14d63c153691" (UID: "c2796e7a-8051-4b7c-80ac-14d63c153691"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:25.315694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.308341    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-provider-credentials\") pod \"c2796e7a-8051-4b7c-80ac-14d63c153691\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") "
Oct 30 21:45:25.315694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.308384    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-plugins-config-volume\") pod \"c2796e7a-8051-4b7c-80ac-14d63c153691\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") "
Oct 30 21:45:25.315694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.308940    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-kube-api-access-js6ld\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-kube-api-access-js6ld\") pod \"c2796e7a-8051-4b7c-80ac-14d63c153691\" (UID: \"c2796e7a-8051-4b7c-80ac-14d63c153691\") "
Oct 30 21:45:25.315694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.310048    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.339050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.337711    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-plugins-config-volume" pod "c2796e7a-8051-4b7c-80ac-14d63c153691" (UID: "c2796e7a-8051-4b7c-80ac-14d63c153691"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:25.340728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.339949    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-provider-credentials" pod "c2796e7a-8051-4b7c-80ac-14d63c153691" (UID: "c2796e7a-8051-4b7c-80ac-14d63c153691"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:25.342638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.342605    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-kube-api-access-js6ld" pod "c2796e7a-8051-4b7c-80ac-14d63c153691" (UID: "c2796e7a-8051-4b7c-80ac-14d63c153691"). InnerVolumeSpecName "kube-api-access-js6ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:25.411992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.411961    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js6ld\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-kube-api-access-js6ld\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.411992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.411990    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2796e7a-8051-4b7c-80ac-14d63c153691-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.412190 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.412002    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2796e7a-8051-4b7c-80ac-14d63c153691-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.941559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.941527    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"
Oct 30 21:45:25.941559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.941535    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr" event={"ID":"c2796e7a-8051-4b7c-80ac-14d63c153691","Type":"ContainerDied","Data":"c798a583ec54b6bdc4b522a739cf7b9549ed2b897c791e1c18fce966c76ca88b"}
Oct 30 21:45:25.942116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.941605    4633 scope.go:122] "RemoveContainer" containerID="5022ee12c2d292e061238e2fe98a0dc3065a6d739a664418018fe94dabc40876"
Oct 30 21:45:25.963817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.963788    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"]
Oct 30 21:45:25.969124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:25.969099    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649fxfwjr"]
Oct 30 21:45:27.645467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.645430    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"]
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646354    4633 cpu_manager.go:414] "containerMap: removing container" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646371    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646381    4633 cpu_manager.go:414] "containerMap: removing container" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646386    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646396    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c2796e7a-8051-4b7c-80ac-14d63c153691" containerName="payload-processing"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646401    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2796e7a-8051-4b7c-80ac-14d63c153691" containerName="payload-processing"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646414    4633 cpu_manager.go:414] "containerMap: removing container" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646419    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646593    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646604    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646615    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2796e7a-8051-4b7c-80ac-14d63c153691" containerName="payload-processing"
Oct 30 21:45:27.648011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.646624    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb72426b-4170-4589-af8d-4f117cb92f2c" containerName="cleanup"
Oct 30 21:45:27.650612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.650593    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.663697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.663650    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e596df-openshift-default-dockercfg-tlppq\""
Oct 30 21:45:27.689042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.689005    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"]
Oct 30 21:45:27.759298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.759200    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-data\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.759599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.759568    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istiod-ca-cert\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.760426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.760294    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-envoy\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.761053 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.761017    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.761429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.761404    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4pw6\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-kube-api-access-h4pw6\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.761604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.761586    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-certs\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.763212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.761754    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-token\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.763785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.763759    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-credential-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.764023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.763995    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-podinfo\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.802809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.802764    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2796e7a-8051-4b7c-80ac-14d63c153691" path="/var/lib/kubelet/pods/c2796e7a-8051-4b7c-80ac-14d63c153691/volumes"
Oct 30 21:45:27.865816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.865781    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-credential-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.865826    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-podinfo\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.865951    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-data\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.865968    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istiod-ca-cert\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866059    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-envoy\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866173    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866292    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4pw6\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-kube-api-access-h4pw6\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866371    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-certs\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866407    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-token\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866415    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866486    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istiod-ca-cert\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866958    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-certs\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.866991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.866964    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-credential-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.867220 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.867188    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-data\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.869332 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.869310    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-envoy\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.869444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.869423    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-podinfo\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.879347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.879320    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-token\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.879567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.879548    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4pw6\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-kube-api-access-h4pw6\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:27.965333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:27.965303    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:28.260795 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:45:28.260761    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c30d58_399a_4f54_b1c5_96ed607bcf89.slice/crio-f30ff144a6583d0f1e824cd9c59fa4b8c4831a29e4f613130e52c0e910bc036e WatchSource:0}: Error finding container f30ff144a6583d0f1e824cd9c59fa4b8c4831a29e4f613130e52c0e910bc036e: Status 404 returned error can't find the container with id f30ff144a6583d0f1e824cd9c59fa4b8c4831a29e4f613130e52c0e910bc036e
Oct 30 21:45:28.262017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:28.261987    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"]
Oct 30 21:45:28.264130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:28.264092    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:28.264221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:28.264163    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:28.264221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:28.264194    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:28.341569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:28.341539    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:45:28.978682 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:28.978625    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" event={"ID":"e4c30d58-399a-4f54-b1c5-96ed607bcf89","Type":"ContainerStarted","Data":"bac570f7876d3f60c92ec9d487a1d37d69194d47fb894e108e1bb18a77d77935"}
Oct 30 21:45:28.979616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:28.979587    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" event={"ID":"e4c30d58-399a-4f54-b1c5-96ed607bcf89","Type":"ContainerStarted","Data":"f30ff144a6583d0f1e824cd9c59fa4b8c4831a29e4f613130e52c0e910bc036e"}
Oct 30 21:45:29.030697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:29.030468    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podStartSLOduration=2.030443828 podStartE2EDuration="2.030443828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:29.027230513 +0000 UTC m=+2637.376957911" watchObservedRunningTime="2026-10-30 21:45:29.030443828 +0000 UTC m=+2637.380171233"
Oct 30 21:45:29.962702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:29.960609    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"]
Oct 30 21:45:29.989723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:29.988532    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.012518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.007925    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-36a099-openshift-default-dockercfg-cv5vg\""
Oct 30 21:45:30.041307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.012826    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 21:45:30.041307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.012878    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 21:45:30.145143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.123552    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.207866    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.207958    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-credential-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.208010    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-podinfo\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.208055    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e5d59e-3940-42c0-906f-38bad87cb8ff-istiod-ca-cert\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.208094    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-envoy\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.208172    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-certs\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.208240    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9hb\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-kube-api-access-pz9hb\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.208330    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-data\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.208766 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.208647    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-token\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310148    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-token\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310260    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310312    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-credential-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310353    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-podinfo\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310393    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e5d59e-3940-42c0-906f-38bad87cb8ff-istiod-ca-cert\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310426    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-envoy\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310493    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-certs\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310567    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9hb\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-kube-api-access-pz9hb\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.311230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.310639    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-data\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.314330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.313943    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"]
Oct 30 21:45:30.316957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.316929    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-certs\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.317734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.317707    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-data\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.318561 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.318537    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e5d59e-3940-42c0-906f-38bad87cb8ff-istiod-ca-cert\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.318992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.318968    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.319615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.319593    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-credential-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.356233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.355722    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-envoy\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.426139 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.425880    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-podinfo\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.448767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.439488    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9hb\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-kube-api-access-pz9hb\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.460249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.460152    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-token\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.734185 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.734106    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:30.971747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.971483    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 21:45:30.971747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:30.971550    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 21:45:31.509657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:31.509183    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"]
Oct 30 21:45:31.572710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:31.572651    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:31.572916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:31.572900    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:31.573010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:31.572998    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:45:31.968875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:31.966844    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 21:45:31.968875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:31.966909    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 21:45:32.595707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:32.584809    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" event={"ID":"07e5d59e-3940-42c0-906f-38bad87cb8ff","Type":"ContainerStarted","Data":"a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb"}
Oct 30 21:45:32.595707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:32.584890    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" event={"ID":"07e5d59e-3940-42c0-906f-38bad87cb8ff","Type":"ContainerStarted","Data":"7027b821056b55e11f44be8cc6eaa47032d0de405acbc100381fc761ca376419"}
Oct 30 21:45:32.653698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:32.652816    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podStartSLOduration=3.652787835 podStartE2EDuration="3.652787835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:32.632382174 +0000 UTC m=+2640.982109588" watchObservedRunningTime="2026-10-30 21:45:32.652787835 +0000 UTC m=+2641.002515232"
Oct 30 21:45:32.739104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:32.736491    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:32.739104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:32.736613    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 21:45:32.739104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:32.736657    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 21:45:32.984982 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:32.984942    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 21:45:32.985311 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:32.985273    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 21:45:33.009960 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:33.009902    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:45:33.742782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:33.742745    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 21:45:33.743412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:33.743379    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 21:45:33.989702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:33.986368    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 21:45:34.020128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:34.020021    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 21:45:34.052705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:34.051910    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:45:34.102102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:34.102068    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:45:34.763702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:34.754724    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 21:45:34.763702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:34.754785    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 21:45:34.991885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:34.986854    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 21:45:34.991885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:34.986920    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 21:45:35.735610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:35.735570    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 21:45:35.735944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:35.735911    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 21:45:35.966486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:35.966047    4633 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body=
Oct 30 21:45:35.966486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:35.966115    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused"
Oct 30 21:45:36.738079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:36.738042    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 21:45:36.738324 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:36.738108    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 21:45:36.971519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:36.971489    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:36.972034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:36.972019    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:36.972759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:36.972739    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:45:37.739363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:37.739327    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 21:45:37.739561 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:37.739386    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 21:45:38.737697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:38.737194    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 21:45:38.737697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:38.737255    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 21:45:39.734871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:39.734833    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 21:45:39.735060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:39.734905    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 21:45:40.740638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:40.740608    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:40.741134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:40.741083    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:40.741808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:45:40.741782    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:45:55.018709 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:45:55.018578    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:46:11.323175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.323138    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"]
Oct 30 21:46:11.345275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.345243    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"]
Oct 30 21:46:11.345461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.345371    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.351582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.351548    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-570f01-openshift-default-dockercfg-vlwqh\""
Oct 30 21:46:11.456596 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.456559    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istiod-ca-cert\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.456803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.456754    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-podinfo\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.456856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.456809    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-data\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.456897 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.456872    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-token\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.457058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.457040    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-credential-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.457105 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.457076    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zzg2\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-kube-api-access-8zzg2\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.457142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.457113    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.457180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.457161    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-envoy\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.457215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.457177    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-certs\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.558712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.558641    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istiod-ca-cert\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.558880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.558818    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-podinfo\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.558950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.558885    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-data\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.558950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.558924    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-token\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559117    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-credential-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559165    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zzg2\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-kube-api-access-8zzg2\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559208    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559250    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-envoy\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559276    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-certs\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559506    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istiod-ca-cert\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559611    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-credential-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559758    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-certs\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559841    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.559955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.559936    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-data\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.562725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.562659    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-envoy\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.563076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.563054    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-podinfo\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.569434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.569411    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-token\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.572000 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.571972    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zzg2\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-kube-api-access-8zzg2\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:11.711816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:11.666660    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:12.319854 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:46:12.319815    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9785510_14ac_4cb6_bde5_b15c4cf827d4.slice/crio-2bcf6d109a5a183d653222a6e0f10d8feabefebd2b75e462eb9834b581c3af28 WatchSource:0}: Error finding container 2bcf6d109a5a183d653222a6e0f10d8feabefebd2b75e462eb9834b581c3af28: Status 404 returned error can't find the container with id 2bcf6d109a5a183d653222a6e0f10d8feabefebd2b75e462eb9834b581c3af28
Oct 30 21:46:12.320393 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:12.320367    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"]
Oct 30 21:46:12.323187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:12.323148    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:46:12.323584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:12.323225    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:46:12.323584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:12.323269    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:46:12.335287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:12.335257    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" event={"ID":"f9785510-14ac-4cb6-bde5-b15c4cf827d4","Type":"ContainerStarted","Data":"2bcf6d109a5a183d653222a6e0f10d8feabefebd2b75e462eb9834b581c3af28"}
Oct 30 21:46:13.371757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:13.371691    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" event={"ID":"f9785510-14ac-4cb6-bde5-b15c4cf827d4","Type":"ContainerStarted","Data":"aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052"}
Oct 30 21:46:13.395124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:13.395064    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" podStartSLOduration=2.395049491 podStartE2EDuration="2.395049491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:13.392409819 +0000 UTC m=+2681.742137216" watchObservedRunningTime="2026-10-30 21:46:13.395049491 +0000 UTC m=+2681.744776958"
Oct 30 21:46:13.684436 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:13.684406    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:13.691119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:13.691095    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:14.382275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:14.382242    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:14.383236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:14.383216    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:46:16.438392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:16.438360    4633 generic.go:356] "Generic (PLEG): container finished" podID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerID="07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385" exitCode=0
Oct 30 21:46:16.439009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:16.438432    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" event={"ID":"a4294be0-dd4b-4442-8f5e-9644df33121c","Type":"ContainerDied","Data":"07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385"}
Oct 30 21:46:16.439009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:16.438934    4633 scope.go:122] "RemoveContainer" containerID="07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385"
Oct 30 21:46:16.440850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:16.440823    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:46:16.440934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:16.440896    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:46:16.440975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:16.440933    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:46:17.678692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:17.670260    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" event={"ID":"a4294be0-dd4b-4442-8f5e-9644df33121c","Type":"ContainerStarted","Data":"e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339"}
Oct 30 21:46:17.937993 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:17.937894    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:46:17.939286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:17.939258    4633 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:46:17.939419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:17.939330    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 21:46:18.837885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:18.837856    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:46:18.940697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:18.939484    4633 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:46:18.940697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:18.939562    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 21:46:19.938178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:19.938139    4633 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:46:19.938602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:19.938200    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 21:46:20.938129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:20.938093    4633 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:46:20.938373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:20.938169    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 21:46:21.939503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:21.939465    4633 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:46:21.940198 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:21.940167    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 21:46:22.940977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:22.940937    4633 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:46:22.941807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:22.941768    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 21:46:23.941389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:23.941356    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:46:23.942730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:23.942710    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:46:25.018336 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:46:25.018298    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:46:33.289164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:33.289128    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:46:33.318364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:33.318335    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:46:33.321417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:33.321391    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:46:33.343010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:33.342987    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:46:36.293976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:36.293865    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:46:43.117626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:43.117593    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"]
Oct 30 21:46:48.118857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.118807    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff" podUID="1fe55c35-e204-4e19-8a12-0f2ece5d70c8" containerName="payload-processing" containerID="cri-o://a0da16d3e350b0cc95c0b82200e06cf52a1420aa2c2cbceecc0a1cb1a8a36320" gracePeriod=25
Oct 30 21:46:48.279385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.279344    4633 generic.go:356] "Generic (PLEG): container finished" podID="1fe55c35-e204-4e19-8a12-0f2ece5d70c8" containerID="a0da16d3e350b0cc95c0b82200e06cf52a1420aa2c2cbceecc0a1cb1a8a36320" exitCode=0
Oct 30 21:46:48.279569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.279416    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff" event={"ID":"1fe55c35-e204-4e19-8a12-0f2ece5d70c8","Type":"ContainerDied","Data":"a0da16d3e350b0cc95c0b82200e06cf52a1420aa2c2cbceecc0a1cb1a8a36320"}
Oct 30 21:46:48.446611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.446586    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:46:48.638513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.638478    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-routing-overlay-volume\") pod \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") "
Oct 30 21:46:48.638739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.638599    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-kube-api-access-gvmd8\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-kube-api-access-gvmd8\") pod \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") "
Oct 30 21:46:48.638866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.638835    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-provider-credentials\") pod \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") "
Oct 30 21:46:48.638936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.638896    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-routing-overlay-volume" pod "1fe55c35-e204-4e19-8a12-0f2ece5d70c8" (UID: "1fe55c35-e204-4e19-8a12-0f2ece5d70c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:48.638995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.638959    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-plugins-config-volume\") pod \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\" (UID: \"1fe55c35-e204-4e19-8a12-0f2ece5d70c8\") "
Oct 30 21:46:48.639454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.639425    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-plugins-config-volume" pod "1fe55c35-e204-4e19-8a12-0f2ece5d70c8" (UID: "1fe55c35-e204-4e19-8a12-0f2ece5d70c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:48.639644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.639474    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:48.641216 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.641193    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-provider-credentials" pod "1fe55c35-e204-4e19-8a12-0f2ece5d70c8" (UID: "1fe55c35-e204-4e19-8a12-0f2ece5d70c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:48.641300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.641230    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-kube-api-access-gvmd8" pod "1fe55c35-e204-4e19-8a12-0f2ece5d70c8" (UID: "1fe55c35-e204-4e19-8a12-0f2ece5d70c8"). InnerVolumeSpecName "kube-api-access-gvmd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:48.741889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.741835    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:48.741889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.741879    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:48.741889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:48.741891    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvmd8\" (UniqueName: \"kubernetes.io/projected/1fe55c35-e204-4e19-8a12-0f2ece5d70c8-kube-api-access-gvmd8\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:49.290621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:49.290593    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"
Oct 30 21:46:49.291107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:49.290627    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff" event={"ID":"1fe55c35-e204-4e19-8a12-0f2ece5d70c8","Type":"ContainerDied","Data":"c6f9560882ef9a35027e26701719bcd71a9f94e8e5658b38d514413ca2c74163"}
Oct 30 21:46:49.291107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:49.290704    4633 scope.go:122] "RemoveContainer" containerID="a0da16d3e350b0cc95c0b82200e06cf52a1420aa2c2cbceecc0a1cb1a8a36320"
Oct 30 21:46:49.321637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:49.321608    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"]
Oct 30 21:46:49.328409 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:49.328379    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-cc67rbgff"]
Oct 30 21:46:49.795940 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:46:49.795901    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe55c35-e204-4e19-8a12-0f2ece5d70c8" path="/var/lib/kubelet/pods/1fe55c35-e204-4e19-8a12-0f2ece5d70c8/volumes"
Oct 30 21:46:55.018593 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:46:55.018555    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:47:07.960964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:07.960927    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"]
Oct 30 21:47:07.962198 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:07.962175    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1fe55c35-e204-4e19-8a12-0f2ece5d70c8" containerName="payload-processing"
Oct 30 21:47:07.962389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:07.962363    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe55c35-e204-4e19-8a12-0f2ece5d70c8" containerName="payload-processing"
Oct 30 21:47:07.962639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:07.962619    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe55c35-e204-4e19-8a12-0f2ece5d70c8" containerName="payload-processing"
Oct 30 21:47:07.980181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:07.980143    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"]
Oct 30 21:47:07.980388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:07.980277    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:07.985829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:07.985638    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3d9d7b\""
Oct 30 21:47:08.017531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:08.017484    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6\" (UID: \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:08.021444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:08.021412    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hswb\" (UniqueName: \"kubernetes.io/projected/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-kube-api-access-2hswb\") pod \"payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6\" (UID: \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:08.134693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:08.134225    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6\" (UID: \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:08.134693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:08.134510    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hswb\" (UniqueName: \"kubernetes.io/projected/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-kube-api-access-2hswb\") pod \"payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6\" (UID: \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:08.219001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:08.217413    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hswb\" (UniqueName: \"kubernetes.io/projected/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-kube-api-access-2hswb\") pod \"payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6\" (UID: \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:08.319708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:08.302163    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6\" (UID: \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:08.669472 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:08.558257    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:10.218931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:10.199299    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"]
Oct 30 21:47:10.637582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:10.637538    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6" event={"ID":"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99","Type":"ContainerStarted","Data":"c18be5d5c910f1b084b9ca8add36314743c958d1919d02e947118fbd1a3e1acb"}
Oct 30 21:47:12.043694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:12.041126    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6" podStartSLOduration=5.041102301 podStartE2EDuration="5.041102301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:12.028985428 +0000 UTC m=+2740.378712847" watchObservedRunningTime="2026-10-30 21:47:12.041102301 +0000 UTC m=+2740.390829700"
Oct 30 21:47:12.126700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:12.126516    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6" event={"ID":"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99","Type":"ContainerStarted","Data":"5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae"}
Oct 30 21:47:13.223704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:13.222750    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:19.749736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.749695    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"]
Oct 30 21:47:19.790123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.790082    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:19.798429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.798393    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8431de\"/\"default-dockercfg-925kp\""
Oct 30 21:47:19.798614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.798586    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8431de\"/\"kube-root-ca.crt\""
Oct 30 21:47:19.799310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.799285    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8431de\"/\"test-model-8431de-kserve-self-signed-certs\""
Oct 30 21:47:19.799740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.799716    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8431de\"/\"openshift-service-ca.crt\""
Oct 30 21:47:19.812060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.812021    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"]
Oct 30 21:47:19.963702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.962031    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-model-cache\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:19.963702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.962351    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-home\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:19.963702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.962387    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tls-certs\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:19.963702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.962418    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxwf\" (UniqueName: \"kubernetes.io/projected/7b0d20f7-3a1a-4875-84fb-29ae05de3304-kube-api-access-6sxwf\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:19.963702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.962614    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tmp-dir\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:19.963702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:19.962748    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-dshm\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.064610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.064511    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-home\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.101692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.097776    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tls-certs\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.101692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.097833    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxwf\" (UniqueName: \"kubernetes.io/projected/7b0d20f7-3a1a-4875-84fb-29ae05de3304-kube-api-access-6sxwf\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.101692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.097924    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tmp-dir\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.101692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.098037    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-dshm\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.101692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.098395    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-model-cache\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.101692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.065441    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-home\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.103252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.103223    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tmp-dir\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.104354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.104330    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-model-cache\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.164044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.163976    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-dshm\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.165322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.165297    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tls-certs\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.166275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.166243    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxwf\" (UniqueName: \"kubernetes.io/projected/7b0d20f7-3a1a-4875-84fb-29ae05de3304-kube-api-access-6sxwf\") pod \"test-model-8431de-kserve-5549fc66b9-njzzj\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") " pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.426631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.426598    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:20.759893 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:20.759864    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"]
Oct 30 21:47:20.761293 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:47:20.761259    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b0d20f7_3a1a_4875_84fb_29ae05de3304.slice/crio-72aae223e15ba7af1e583d48395dc632157b6e22272f6a282f4bfbc3c7c741d0 WatchSource:0}: Error finding container 72aae223e15ba7af1e583d48395dc632157b6e22272f6a282f4bfbc3c7c741d0: Status 404 returned error can't find the container with id 72aae223e15ba7af1e583d48395dc632157b6e22272f6a282f4bfbc3c7c741d0
Oct 30 21:47:21.446464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:21.446431    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj" event={"ID":"7b0d20f7-3a1a-4875-84fb-29ae05de3304","Type":"ContainerStarted","Data":"cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73"}
Oct 30 21:47:21.446464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:21.446473    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj" event={"ID":"7b0d20f7-3a1a-4875-84fb-29ae05de3304","Type":"ContainerStarted","Data":"72aae223e15ba7af1e583d48395dc632157b6e22272f6a282f4bfbc3c7c741d0"}
Oct 30 21:47:21.684577 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:21.468612    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj" podStartSLOduration=2.468597702 podStartE2EDuration="2.468597702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:21.466263417 +0000 UTC m=+2749.815990813" watchObservedRunningTime="2026-10-30 21:47:21.468597702 +0000 UTC m=+2749.818325097"
Oct 30 21:47:22.426797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:22.426761    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:22.460791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:22.460751    4633 generic.go:356] "Generic (PLEG): container finished" podID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerID="aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052" exitCode=0
Oct 30 21:47:22.460988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:22.460791    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" event={"ID":"f9785510-14ac-4cb6-bde5-b15c4cf827d4","Type":"ContainerDied","Data":"aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052"}
Oct 30 21:47:22.461426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:22.461391    4633 scope.go:122] "RemoveContainer" containerID="aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052"
Oct 30 21:47:22.464459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:22.464423    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:47:22.464566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:22.464504    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:47:22.464566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:22.464548    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:47:23.473101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:23.473066    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" event={"ID":"f9785510-14ac-4cb6-bde5-b15c4cf827d4","Type":"ContainerStarted","Data":"e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162"}
Oct 30 21:47:23.684895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:23.684856    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:47:23.685089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:23.685050    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 21:47:23.685161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:23.685120    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 21:47:24.301528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:24.301498    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:47:24.684999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:24.684963    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 21:47:24.685415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:24.685033    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 21:47:24.870330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:24.870297    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"]
Oct 30 21:47:25.051845 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:47:25.051745    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:47:25.684789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:25.684753    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 21:47:25.685002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:25.684816    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 21:47:26.684439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:26.684406    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 21:47:26.684863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:26.684458    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 21:47:27.685869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:27.685806    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 21:47:27.698700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:27.686478    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 21:47:28.684966 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:28.684923    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 21:47:28.685278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:28.684998    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 21:47:29.382280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:29.382250    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:47:29.690920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:29.690887    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:47:29.752305 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:29.752279    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:47:29.870698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:29.870631    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns" podUID="e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" containerName="payload-pre-processing" containerID="cri-o://3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459" gracePeriod=25
Oct 30 21:47:30.426923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:30.426886    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:30.440619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:30.440586    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:30.771352 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:30.771267    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:47:39.572152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:39.572108    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns" podUID="e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:47:46.547385 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.547346    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"]
Oct 30 21:47:46.548147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.547638    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq" podUID="61147a40-6652-4743-ab47-9e9c49a4b880" containerName="maas-api" containerID="cri-o://2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac" gracePeriod=30
Oct 30 21:47:46.868530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.868505    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:47:46.908057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.908019    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-metrics-certs\") pod \"61147a40-6652-4743-ab47-9e9c49a4b880\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") "
Oct 30 21:47:46.908395 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.908370    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61147a40-6652-4743-ab47-9e9c49a4b880-kube-api-access-6dg86\" (UniqueName: \"kubernetes.io/projected/61147a40-6652-4743-ab47-9e9c49a4b880-kube-api-access-6dg86\") pod \"61147a40-6652-4743-ab47-9e9c49a4b880\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") "
Oct 30 21:47:46.908948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.908919    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61147a40-6652-4743-ab47-9e9c49a4b880-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61147a40-6652-4743-ab47-9e9c49a4b880-openshift-service-ca\") pod \"61147a40-6652-4743-ab47-9e9c49a4b880\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") "
Oct 30 21:47:46.909151 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.909122    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-maas-api-tls\") pod \"61147a40-6652-4743-ab47-9e9c49a4b880\" (UID: \"61147a40-6652-4743-ab47-9e9c49a4b880\") "
Oct 30 21:47:46.911780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.911750    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-metrics-certs" pod "61147a40-6652-4743-ab47-9e9c49a4b880" (UID: "61147a40-6652-4743-ab47-9e9c49a4b880"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:46.912122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.912074    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61147a40-6652-4743-ab47-9e9c49a4b880-openshift-service-ca" pod "61147a40-6652-4743-ab47-9e9c49a4b880" (UID: "61147a40-6652-4743-ab47-9e9c49a4b880"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:46.914106 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.914079    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-maas-api-tls" pod "61147a40-6652-4743-ab47-9e9c49a4b880" (UID: "61147a40-6652-4743-ab47-9e9c49a4b880"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:46.914219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:46.914180    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61147a40-6652-4743-ab47-9e9c49a4b880-kube-api-access-6dg86" pod "61147a40-6652-4743-ab47-9e9c49a4b880" (UID: "61147a40-6652-4743-ab47-9e9c49a4b880"). InnerVolumeSpecName "kube-api-access-6dg86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:47.011772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.011735    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:47.011772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.011769    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dg86\" (UniqueName: \"kubernetes.io/projected/61147a40-6652-4743-ab47-9e9c49a4b880-kube-api-access-6dg86\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:47.011995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.011786    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61147a40-6652-4743-ab47-9e9c49a4b880-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:47.011995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.011800    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61147a40-6652-4743-ab47-9e9c49a4b880-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:47.348728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.345576    4633 generic.go:356] "Generic (PLEG): container finished" podID="61147a40-6652-4743-ab47-9e9c49a4b880" containerID="2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac" exitCode=0
Oct 30 21:47:47.348728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.346034    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq" event={"ID":"61147a40-6652-4743-ab47-9e9c49a4b880","Type":"ContainerDied","Data":"2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac"}
Oct 30 21:47:47.348728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.346099    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq" event={"ID":"61147a40-6652-4743-ab47-9e9c49a4b880","Type":"ContainerDied","Data":"a8c93a2cf8aeb368f9f56114fb196672d903bf7517c4677aafd6fa7d047eda42"}
Oct 30 21:47:47.348728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.346131    4633 scope.go:122] "RemoveContainer" containerID="2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac"
Oct 30 21:47:47.348728 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.346330    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"
Oct 30 21:47:47.429002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.428966    4633 scope.go:122] "RemoveContainer" containerID="2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac"
Oct 30 21:47:47.431569 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:47:47.431469    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac\": container with ID starting with 2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac not found: ID does not exist" containerID="2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac"
Oct 30 21:47:47.431569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.431517    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac"} err="failed to get container status \"2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac\": rpc error: code = NotFound desc = could not find container \"2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac\": container with ID starting with 2e6d81100c0af597a9560365b55b3e40a677d964f10498141f7353e6ca38ceac not found: ID does not exist"
Oct 30 21:47:47.433737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.433709    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"]
Oct 30 21:47:47.441914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.441879    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7b5a7c-6dcd75886d-smhzq"]
Oct 30 21:47:47.793662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:47.793618    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61147a40-6652-4743-ab47-9e9c49a4b880" path="/var/lib/kubelet/pods/61147a40-6652-4743-ab47-9e9c49a4b880/volumes"
Oct 30 21:47:48.134619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.134539    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"]
Oct 30 21:47:48.136127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.136104    4633 cpu_manager.go:414] "containerMap: removing container" podUID="61147a40-6652-4743-ab47-9e9c49a4b880" containerName="maas-api"
Oct 30 21:47:48.136127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.136128    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61147a40-6652-4743-ab47-9e9c49a4b880" containerName="maas-api"
Oct 30 21:47:48.136310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.136283    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="61147a40-6652-4743-ab47-9e9c49a4b880" containerName="maas-api"
Oct 30 21:47:48.141756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.141733    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:48.165566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.165533    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"]
Oct 30 21:47:48.241109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.241081    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-87ae68\""
Oct 30 21:47:48.242954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.242931    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8f84c7-6449-4add-be75-f2e3819e04a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl\" (UID: \"5b8f84c7-6449-4add-be75-f2e3819e04a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:48.243134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.243119    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjb9h\" (UniqueName: \"kubernetes.io/projected/5b8f84c7-6449-4add-be75-f2e3819e04a7-kube-api-access-xjb9h\") pod \"payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl\" (UID: \"5b8f84c7-6449-4add-be75-f2e3819e04a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:48.399763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.394848    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8f84c7-6449-4add-be75-f2e3819e04a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl\" (UID: \"5b8f84c7-6449-4add-be75-f2e3819e04a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:48.399763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.395119    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjb9h\" (UniqueName: \"kubernetes.io/projected/5b8f84c7-6449-4add-be75-f2e3819e04a7-kube-api-access-xjb9h\") pod \"payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl\" (UID: \"5b8f84c7-6449-4add-be75-f2e3819e04a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:48.399763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.396660    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8f84c7-6449-4add-be75-f2e3819e04a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl\" (UID: \"5b8f84c7-6449-4add-be75-f2e3819e04a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:48.457709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.457620    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjb9h\" (UniqueName: \"kubernetes.io/projected/5b8f84c7-6449-4add-be75-f2e3819e04a7-kube-api-access-xjb9h\") pod \"payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl\" (UID: \"5b8f84c7-6449-4add-be75-f2e3819e04a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:48.474070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:48.465974    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:49.052861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:49.052820    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"]
Oct 30 21:47:49.493438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:49.493393    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl" event={"ID":"5b8f84c7-6449-4add-be75-f2e3819e04a7","Type":"ContainerStarted","Data":"a574a24bc46c51fade833df30555b55d2aac37e026ffb0327e0e25e3666d4f46"}
Oct 30 21:47:49.493650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:49.493493    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl" event={"ID":"5b8f84c7-6449-4add-be75-f2e3819e04a7","Type":"ContainerStarted","Data":"8390fcbd572789a210d084d6bbbfb1f2e4fbf910d2007e5520350cb87a91ed11"}
Oct 30 21:47:49.494900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:49.494875    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:47:49.523129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:49.523077    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl" podStartSLOduration=1.523063022 podStartE2EDuration="1.523063022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:49.517767188 +0000 UTC m=+2777.867494589" watchObservedRunningTime="2026-10-30 21:47:49.523063022 +0000 UTC m=+2777.872790417"
Oct 30 21:47:49.572074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:49.572034    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns" podUID="e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:47:50.191478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.191449    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:47:50.375769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.375682    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-kube-api-access-4txrn\" (UniqueName: \"kubernetes.io/projected/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-kube-api-access-4txrn\") pod \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\" (UID: \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\") "
Oct 30 21:47:50.375769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.375755    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-plugins-config-volume\") pod \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\" (UID: \"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7\") "
Oct 30 21:47:50.377189 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.377158    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-plugins-config-volume" pod "e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" (UID: "e9ce2b2c-c62b-474d-8af8-4350bb1ddee7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:50.379388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.379356    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-kube-api-access-4txrn" pod "e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" (UID: "e9ce2b2c-c62b-474d-8af8-4350bb1ddee7"). InnerVolumeSpecName "kube-api-access-4txrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:50.477965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.477879    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4txrn\" (UniqueName: \"kubernetes.io/projected/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-kube-api-access-4txrn\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:50.477965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.477909    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:50.503637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.503599    4633 generic.go:356] "Generic (PLEG): container finished" podID="e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" containerID="3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459" exitCode=0
Oct 30 21:47:50.503872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.503651    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"
Oct 30 21:47:50.503872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.503714    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns" event={"ID":"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7","Type":"ContainerDied","Data":"3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459"}
Oct 30 21:47:50.503872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.503784    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns" event={"ID":"e9ce2b2c-c62b-474d-8af8-4350bb1ddee7","Type":"ContainerDied","Data":"afcafee578fb842d73cbadca9cb5d4ebd4ad83685743a6639c31e5a60ea28a40"}
Oct 30 21:47:50.503872 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.503814    4633 scope.go:122] "RemoveContainer" containerID="3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459"
Oct 30 21:47:50.518031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.518010    4633 scope.go:122] "RemoveContainer" containerID="3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459"
Oct 30 21:47:50.518396 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:47:50.518371    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459\": container with ID starting with 3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459 not found: ID does not exist" containerID="3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459"
Oct 30 21:47:50.518487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.518409    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459"} err="failed to get container status \"3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459\": rpc error: code = NotFound desc = could not find container \"3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459\": container with ID starting with 3897f1e40a0c28e1297581c6e188ee8abc4dcef740c41c8c7dae298298729459 not found: ID does not exist"
Oct 30 21:47:50.532828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.532792    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"]
Oct 30 21:47:50.538386 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:50.538363    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7b5a7c-659db5d9c9-c29ns"]
Oct 30 21:47:51.519772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:51.519726    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:47:52.037240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:52.034471    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" path="/var/lib/kubelet/pods/e9ce2b2c-c62b-474d-8af8-4350bb1ddee7/volumes"
Oct 30 21:47:55.105597 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:47:55.105566    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:47:55.308423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:55.308385    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"]
Oct 30 21:47:55.308950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:55.308657    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" containerID="cri-o://801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601" gracePeriod=30
Oct 30 21:47:55.865261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:55.863862    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:47:55.865261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:47:55.863934    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:48:00.743777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.743749    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:48:00.829817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.829783    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9e4e393-82cf-44c5-9247-321be52321a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e4e393-82cf-44c5-9247-321be52321a3-istio-podinfo\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.830008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.829828    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-credential-socket\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.830008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.829984    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-certs\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.830147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.830097    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e4e393-82cf-44c5-9247-321be52321a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e4e393-82cf-44c5-9247-321be52321a3-istiod-ca-cert\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.830333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.830309    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-certs" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:00.830416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.830330    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-credential-socket" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:00.830473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.830450    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e4e393-82cf-44c5-9247-321be52321a3-istiod-ca-cert" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:00.830539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.830521    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-kube-api-access-kmg94\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-kube-api-access-kmg94\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.830760 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.830707    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-data\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.830851 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.830836    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-socket\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.830939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.830919    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-istio-token\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-istio-token\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.831315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.831063    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-data" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:00.831416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.831187    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-socket" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:00.832719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.831551    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-envoy\") pod \"a9e4e393-82cf-44c5-9247-321be52321a3\" (UID: \"a9e4e393-82cf-44c5-9247-321be52321a3\") "
Oct 30 21:48:00.832719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.832432    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.832719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.832451    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.832719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.832460    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9e4e393-82cf-44c5-9247-321be52321a3-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.832719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.832471    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.832719 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.832479    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.833338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.833314    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-istio-token" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:00.833442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.833428    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9e4e393-82cf-44c5-9247-321be52321a3-istio-podinfo" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:48:00.833641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.833614    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-kube-api-access-kmg94" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "kube-api-access-kmg94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:00.835202 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.835182    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-envoy" pod "a9e4e393-82cf-44c5-9247-321be52321a3" (UID: "a9e4e393-82cf-44c5-9247-321be52321a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:00.934880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.934838    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.934880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.934870    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9e4e393-82cf-44c5-9247-321be52321a3-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.935091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.934892    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9e4e393-82cf-44c5-9247-321be52321a3-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:00.935091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:00.934906    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmg94\" (UniqueName: \"kubernetes.io/projected/a9e4e393-82cf-44c5-9247-321be52321a3-kube-api-access-kmg94\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.017721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.017594    4633 generic.go:356] "Generic (PLEG): container finished" podID="a9e4e393-82cf-44c5-9247-321be52321a3" containerID="801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601" exitCode=0
Oct 30 21:48:01.017721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.017637    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" event={"ID":"a9e4e393-82cf-44c5-9247-321be52321a3","Type":"ContainerDied","Data":"801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601"}
Oct 30 21:48:01.017721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.017648    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"
Oct 30 21:48:01.017721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.017704    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7" event={"ID":"a9e4e393-82cf-44c5-9247-321be52321a3","Type":"ContainerDied","Data":"1f48cc4607dca5c509c7bb3167e212a8210ac987b81d63fed27eefa04ad54aab"}
Oct 30 21:48:01.018011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.017733    4633 scope.go:122] "RemoveContainer" containerID="801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601"
Oct 30 21:48:01.033991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.033972    4633 scope.go:122] "RemoveContainer" containerID="272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0"
Oct 30 21:48:01.046169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.046148    4633 scope.go:122] "RemoveContainer" containerID="801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601"
Oct 30 21:48:01.046487 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:01.046467    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601\": container with ID starting with 801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601 not found: ID does not exist" containerID="801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601"
Oct 30 21:48:01.046558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.046497    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601"} err="failed to get container status \"801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601\": rpc error: code = NotFound desc = could not find container \"801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601\": container with ID starting with 801535a52ea48b2813945dcf95908f3bc44df569d0d68f69be1f6d4aa5d74601 not found: ID does not exist"
Oct 30 21:48:01.046558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.046523    4633 scope.go:122] "RemoveContainer" containerID="272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0"
Oct 30 21:48:01.046846 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:01.046823    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0\": container with ID starting with 272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0 not found: ID does not exist" containerID="272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0"
Oct 30 21:48:01.046919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.046852    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0"} err="failed to get container status \"272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0\": rpc error: code = NotFound desc = could not find container \"272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0\": container with ID starting with 272f9055124d90f3666c934caefb80fe950849ba6d05da4687d0bd6f2215a6d0 not found: ID does not exist"
Oct 30 21:48:01.051018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.050992    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"]
Oct 30 21:48:01.070974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.070943    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-tdcd7"]
Oct 30 21:48:01.521567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.521539    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:48:01.794045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:01.793967    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" path="/var/lib/kubelet/pods/a9e4e393-82cf-44c5-9247-321be52321a3/volumes"
Oct 30 21:48:06.122210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.122174    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"]
Oct 30 21:48:06.140798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.140763    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"]
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.744754    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"]
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.746876    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" containerName="payload-pre-processing"
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.746904    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" containerName="payload-pre-processing"
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.746955    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy"
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.746964    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy"
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.747022    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy"
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.747031    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy"
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.747495    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy"
Oct 30 21:48:06.748692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.747546    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ce2b2c-c62b-474d-8af8-4350bb1ddee7" containerName="payload-pre-processing"
Oct 30 21:48:06.860764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.859880    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.865972 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.865943    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cf51d4-openshift-default-dockercfg-jq2sn\""
Oct 30 21:48:06.887736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.887706    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"]
Oct 30 21:48:06.890159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.890119    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e4e393-82cf-44c5-9247-321be52321a3" containerName="istio-proxy"
Oct 30 21:48:06.894995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.894971    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.897746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.897724    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"]
Oct 30 21:48:06.904799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.904777    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"]
Oct 30 21:48:06.981477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.981441    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.981749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.981537    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.981876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.981750    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.981876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.981848    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.981969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.981935    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbdx\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-kube-api-access-vjbdx\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.982021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.981970    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.982021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.981999    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.982100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982075    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.982141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982119    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.982233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982213    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.982270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982246    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.982270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982262    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.982384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982372    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.982451 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982436    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7zs\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-kube-api-access-pf7zs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.982530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982513    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.982583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982537    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:06.982583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982556    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:06.982733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:06.982591    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.095576    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.095718    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.095791    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbdx\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-kube-api-access-vjbdx\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.095838    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.095880    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.095909    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.095971    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096058    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096116    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096147    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096237    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096337    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7zs\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-kube-api-access-pf7zs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096408    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096431    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096454    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096507    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096565    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.096592    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.097160    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.099770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.097427    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.111979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.106470    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.116058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.115988    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.138557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.131056    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.138557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.133106    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.138557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.133348    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.138557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.133948    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.138557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.134757    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.138557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.135113    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.138557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.135713    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.163837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.135940    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.164039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.164013    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.171823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.171789    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.174836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.174807    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7zs\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-kube-api-access-pf7zs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.175280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.175258    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbdx\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-kube-api-access-vjbdx\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.175725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.175703    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.178964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.178426    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.187648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.187625    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:07.216283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.216258    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:07.491862 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:48:07.491831    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c14236_fd6a_404b_8219_5ff6ba2f98f3.slice/crio-f8d6f007e2de60cb35edb8fb3da9c1d567e65c731124b0b4df61e066cb7999a8 WatchSource:0}: Error finding container f8d6f007e2de60cb35edb8fb3da9c1d567e65c731124b0b4df61e066cb7999a8: Status 404 returned error can't find the container with id f8d6f007e2de60cb35edb8fb3da9c1d567e65c731124b0b4df61e066cb7999a8
Oct 30 21:48:07.493369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.493342    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"]
Oct 30 21:48:07.494881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.494850    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:07.494944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.494913    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:07.494981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.494947    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:07.544743 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:48:07.544708    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244ce12a_d4e5_4ee7_9b57_14b1f13f345b.slice/crio-3552e58088127aeeb451483dddb84babc44a38254f6fa28d45304759076865f2 WatchSource:0}: Error finding container 3552e58088127aeeb451483dddb84babc44a38254f6fa28d45304759076865f2: Status 404 returned error can't find the container with id 3552e58088127aeeb451483dddb84babc44a38254f6fa28d45304759076865f2
Oct 30 21:48:07.545826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.545791    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"]
Oct 30 21:48:07.549912 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.549871    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:07.550017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.549969    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:07.550090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:07.550017    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:08.193212 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.193176    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" event={"ID":"e6c14236-fd6a-404b-8219-5ff6ba2f98f3","Type":"ContainerStarted","Data":"e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165"}
Oct 30 21:48:08.196764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.196737    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" event={"ID":"e6c14236-fd6a-404b-8219-5ff6ba2f98f3","Type":"ContainerStarted","Data":"f8d6f007e2de60cb35edb8fb3da9c1d567e65c731124b0b4df61e066cb7999a8"}
Oct 30 21:48:08.198411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.198389    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" event={"ID":"244ce12a-d4e5-4ee7-9b57-14b1f13f345b","Type":"ContainerStarted","Data":"ade44335502cab19cd3da79450c2e7f19f8e417cbfd840dd6ac336eb66a109a2"}
Oct 30 21:48:08.199435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.199416    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" event={"ID":"244ce12a-d4e5-4ee7-9b57-14b1f13f345b","Type":"ContainerStarted","Data":"3552e58088127aeeb451483dddb84babc44a38254f6fa28d45304759076865f2"}
Oct 30 21:48:08.217791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.217759    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:08.229777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.229720    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podStartSLOduration=2.229700771 podStartE2EDuration="2.229700771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:08.221915099 +0000 UTC m=+2796.571642496" watchObservedRunningTime="2026-10-30 21:48:08.229700771 +0000 UTC m=+2796.579428168"
Oct 30 21:48:08.231846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.231815    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 21:48:08.232099 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.232075    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 21:48:08.250242 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:08.250194    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podStartSLOduration=2.250180405 podStartE2EDuration="2.250180405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:08.244329951 +0000 UTC m=+2796.594057347" watchObservedRunningTime="2026-10-30 21:48:08.250180405 +0000 UTC m=+2796.599907800"
Oct 30 21:48:09.192046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:09.191842    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:09.192885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:09.192852    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 21:48:09.193138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:09.192904    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 21:48:09.232695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:09.222165    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 21:48:09.232695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:09.222230    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 21:48:10.190545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:10.190504    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 21:48:10.190795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:10.190562    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 21:48:10.218120 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:10.218079    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 21:48:10.221800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:10.221750    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 21:48:11.125709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:11.124746    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d" podUID="05e11b5a-7968-4403-b6aa-2d83dcbff14b" containerName="payload-processing" containerID="cri-o://dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326" gracePeriod=25
Oct 30 21:48:11.149254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:11.147862    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9" podUID="b905e0bb-b295-4752-b8d1-19ae58096f46" containerName="payload-pre-processing" containerID="cri-o://44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb" gracePeriod=25
Oct 30 21:48:11.189192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:11.189140    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 21:48:11.189524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:11.189493    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 21:48:11.225744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:11.219283    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 21:48:11.225744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:11.219353    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 21:48:12.188285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:12.188239    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 21:48:12.188800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:12.188294    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 21:48:12.225700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:12.225209    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 21:48:12.225700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:12.225289    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 21:48:13.244709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:13.243563    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 21:48:13.244709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:13.243633    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 21:48:13.244709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:13.244170    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 21:48:13.244709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:13.244220    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 21:48:14.192697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:14.189093    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 21:48:14.192697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:14.189184    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 21:48:14.230730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:14.229849    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 21:48:14.230730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:14.229915    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 21:48:15.196704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:15.192059    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 30 21:48:15.196704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:15.192141    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 30 21:48:15.223702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:15.219873    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 30 21:48:15.223702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:15.219937    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 30 21:48:16.193114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:16.193082    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:16.193635 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:16.193606    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:16.194355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:16.194330    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:48:16.222609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:16.222580    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:16.223116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:16.223094    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:16.223996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:16.223977    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:48:18.435361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:18.435325    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9" podUID="b905e0bb-b295-4752-b8d1-19ae58096f46" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:48:18.435888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:18.435852    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d" podUID="05e11b5a-7968-4403-b6aa-2d83dcbff14b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:48:23.571886 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:23.571847    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"]
Oct 30 21:48:23.572429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:23.572096    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz" podUID="ffd17c96-6623-4d56-b849-b4a94e75de97" containerName="maas-api" containerID="cri-o://4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0" gracePeriod=30
Oct 30 21:48:24.330798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.330769    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:48:24.364361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.364327    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"]
Oct 30 21:48:24.365224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.365207    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ffd17c96-6623-4d56-b849-b4a94e75de97" containerName="maas-api"
Oct 30 21:48:24.365271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.365228    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd17c96-6623-4d56-b849-b4a94e75de97" containerName="maas-api"
Oct 30 21:48:24.365432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.365423    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd17c96-6623-4d56-b849-b4a94e75de97" containerName="maas-api"
Oct 30 21:48:24.369239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.369219    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.377968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.377942    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-87ae68-serving-cert\""
Oct 30 21:48:24.379512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.379482    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-metrics-certs\") pod \"ffd17c96-6623-4d56-b849-b4a94e75de97\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") "
Oct 30 21:48:24.379647 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.379530    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls\") pod \"ffd17c96-6623-4d56-b849-b4a94e75de97\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") "
Oct 30 21:48:24.379836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.379819    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd17c96-6623-4d56-b849-b4a94e75de97-kube-api-access-ln8cq\" (UniqueName: \"kubernetes.io/projected/ffd17c96-6623-4d56-b849-b4a94e75de97-kube-api-access-ln8cq\") pod \"ffd17c96-6623-4d56-b849-b4a94e75de97\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") "
Oct 30 21:48:24.379911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.379885    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd17c96-6623-4d56-b849-b4a94e75de97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd17c96-6623-4d56-b849-b4a94e75de97-openshift-service-ca\") pod \"ffd17c96-6623-4d56-b849-b4a94e75de97\" (UID: \"ffd17c96-6623-4d56-b849-b4a94e75de97\") "
Oct 30 21:48:24.380343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.380316    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.380450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.380407    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-metrics-certs\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.380626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.380607    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwlzv\" (UniqueName: \"kubernetes.io/projected/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-kube-api-access-bwlzv\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.380742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.380643    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-maas-api-tls\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.381091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.381069    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd17c96-6623-4d56-b849-b4a94e75de97-openshift-service-ca" pod "ffd17c96-6623-4d56-b849-b4a94e75de97" (UID: "ffd17c96-6623-4d56-b849-b4a94e75de97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:24.381808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.381780    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"]
Oct 30 21:48:24.382939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.382910    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls" pod "ffd17c96-6623-4d56-b849-b4a94e75de97" (UID: "ffd17c96-6623-4d56-b849-b4a94e75de97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:24.383611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.383590    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-metrics-certs" pod "ffd17c96-6623-4d56-b849-b4a94e75de97" (UID: "ffd17c96-6623-4d56-b849-b4a94e75de97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:24.385095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.385064    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd17c96-6623-4d56-b849-b4a94e75de97-kube-api-access-ln8cq" pod "ffd17c96-6623-4d56-b849-b4a94e75de97" (UID: "ffd17c96-6623-4d56-b849-b4a94e75de97"). InnerVolumeSpecName "kube-api-access-ln8cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:24.532590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.501609    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.533067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.533029    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-metrics-certs\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.534091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.514027    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.534218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.534099    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwlzv\" (UniqueName: \"kubernetes.io/projected/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-kube-api-access-bwlzv\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.534218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.534140    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-maas-api-tls\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.549695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.534349    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:24.549695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.534369    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd17c96-6623-4d56-b849-b4a94e75de97-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:24.549695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.534385    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8cq\" (UniqueName: \"kubernetes.io/projected/ffd17c96-6623-4d56-b849-b4a94e75de97-kube-api-access-ln8cq\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:24.549695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.534399    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd17c96-6623-4d56-b849-b4a94e75de97-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:24.549695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.544925    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-metrics-certs\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.560812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.559721    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwlzv\" (UniqueName: \"kubernetes.io/projected/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-kube-api-access-bwlzv\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.561064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.560868    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-maas-api-tls\") pod \"maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.689010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.688947    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:24.691445 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.691414    4633 generic.go:356] "Generic (PLEG): container finished" podID="ffd17c96-6623-4d56-b849-b4a94e75de97" containerID="4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0" exitCode=0
Oct 30 21:48:24.691619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.691481    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"
Oct 30 21:48:24.691619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.691505    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz" event={"ID":"ffd17c96-6623-4d56-b849-b4a94e75de97","Type":"ContainerDied","Data":"4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0"}
Oct 30 21:48:24.691619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.691567    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz" event={"ID":"ffd17c96-6623-4d56-b849-b4a94e75de97","Type":"ContainerDied","Data":"46ea2254f689c36bf73ce146d561d8ad44ae4a186340f45c945fce1b17c85f98"}
Oct 30 21:48:24.691619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.691594    4633 scope.go:122] "RemoveContainer" containerID="4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0"
Oct 30 21:48:24.717129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.717099    4633 scope.go:122] "RemoveContainer" containerID="4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0"
Oct 30 21:48:24.717509 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:24.717485    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0\": container with ID starting with 4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0 not found: ID does not exist" containerID="4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0"
Oct 30 21:48:24.717594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.717519    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0"} err="failed to get container status \"4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0\": rpc error: code = NotFound desc = could not find container \"4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0\": container with ID starting with 4412e1dbd4b9679528aab1294abf006d3ad98b78c5f77300f1daa5345cf4bca0 not found: ID does not exist"
Oct 30 21:48:24.727024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.726990    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"]
Oct 30 21:48:24.732844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:24.732815    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b30a42-769bc87cb-cfkhz"]
Oct 30 21:48:25.014888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:25.014864    4633 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:48:25.017750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:25.017710    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"]
Oct 30 21:48:25.024334 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:25.024304    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:48:25.705879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:25.705841    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf" event={"ID":"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e","Type":"ContainerStarted","Data":"39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b"}
Oct 30 21:48:25.706381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:25.705901    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf" event={"ID":"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e","Type":"ContainerStarted","Data":"9d4e32cb8aeb0104187a73127468f3e47f55d67de71bd8c8cda72f6cfd2c78e5"}
Oct 30 21:48:25.706381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:25.705955    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:25.738003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:25.737950    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf" podStartSLOduration=7.248942704 podStartE2EDuration="7.737934952s" totalImagesPullingTime="488.992248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:25.734874588 +0000 UTC m=+2814.084602007" watchObservedRunningTime="2026-10-30 21:48:25.737934952 +0000 UTC m=+2814.087662347"
Oct 30 21:48:25.799491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:25.799400    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd17c96-6623-4d56-b849-b4a94e75de97" path="/var/lib/kubelet/pods/ffd17c96-6623-4d56-b849-b4a94e75de97/volumes"
Oct 30 21:48:28.428294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:28.428258    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d" podUID="05e11b5a-7968-4403-b6aa-2d83dcbff14b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:48:28.428805 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:28.428265    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9" podUID="b905e0bb-b295-4752-b8d1-19ae58096f46" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:48:31.406016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.405984    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:48:31.446791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.446765    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:48:31.513975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.513937    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b905e0bb-b295-4752-b8d1-19ae58096f46-kube-api-access-p69f8\" (UniqueName: \"kubernetes.io/projected/b905e0bb-b295-4752-b8d1-19ae58096f46-kube-api-access-p69f8\") pod \"b905e0bb-b295-4752-b8d1-19ae58096f46\" (UID: \"b905e0bb-b295-4752-b8d1-19ae58096f46\") "
Oct 30 21:48:31.514297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.514037    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b905e0bb-b295-4752-b8d1-19ae58096f46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b905e0bb-b295-4752-b8d1-19ae58096f46-plugins-config-volume\") pod \"b905e0bb-b295-4752-b8d1-19ae58096f46\" (UID: \"b905e0bb-b295-4752-b8d1-19ae58096f46\") "
Oct 30 21:48:31.514297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.514274    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e11b5a-7968-4403-b6aa-2d83dcbff14b-kube-api-access-bj2rt\" (UniqueName: \"kubernetes.io/projected/05e11b5a-7968-4403-b6aa-2d83dcbff14b-kube-api-access-bj2rt\") pod \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\" (UID: \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\") "
Oct 30 21:48:31.514515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.514312    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e11b5a-7968-4403-b6aa-2d83dcbff14b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e11b5a-7968-4403-b6aa-2d83dcbff14b-plugins-config-volume\") pod \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\" (UID: \"05e11b5a-7968-4403-b6aa-2d83dcbff14b\") "
Oct 30 21:48:31.514515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.514462    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b905e0bb-b295-4752-b8d1-19ae58096f46-plugins-config-volume" pod "b905e0bb-b295-4752-b8d1-19ae58096f46" (UID: "b905e0bb-b295-4752-b8d1-19ae58096f46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:31.514775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.514745    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e11b5a-7968-4403-b6aa-2d83dcbff14b-plugins-config-volume" pod "05e11b5a-7968-4403-b6aa-2d83dcbff14b" (UID: "05e11b5a-7968-4403-b6aa-2d83dcbff14b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:31.515206 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.515189    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05e11b5a-7968-4403-b6aa-2d83dcbff14b-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:31.515269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.515209    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b905e0bb-b295-4752-b8d1-19ae58096f46-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:31.516580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.516551    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e11b5a-7968-4403-b6aa-2d83dcbff14b-kube-api-access-bj2rt" pod "05e11b5a-7968-4403-b6aa-2d83dcbff14b" (UID: "05e11b5a-7968-4403-b6aa-2d83dcbff14b"). InnerVolumeSpecName "kube-api-access-bj2rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:31.516986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.516962    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b905e0bb-b295-4752-b8d1-19ae58096f46-kube-api-access-p69f8" pod "b905e0bb-b295-4752-b8d1-19ae58096f46" (UID: "b905e0bb-b295-4752-b8d1-19ae58096f46"). InnerVolumeSpecName "kube-api-access-p69f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:31.617390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.617351    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p69f8\" (UniqueName: \"kubernetes.io/projected/b905e0bb-b295-4752-b8d1-19ae58096f46-kube-api-access-p69f8\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:31.617390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.617383    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2rt\" (UniqueName: \"kubernetes.io/projected/05e11b5a-7968-4403-b6aa-2d83dcbff14b-kube-api-access-bj2rt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:31.723520 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.723451    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:48:31.776864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.776829    4633 generic.go:356] "Generic (PLEG): container finished" podID="05e11b5a-7968-4403-b6aa-2d83dcbff14b" containerID="dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326" exitCode=0
Oct 30 21:48:31.777068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.776872    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"
Oct 30 21:48:31.777068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.776905    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d" event={"ID":"05e11b5a-7968-4403-b6aa-2d83dcbff14b","Type":"ContainerDied","Data":"dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326"}
Oct 30 21:48:31.777068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.776959    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d" event={"ID":"05e11b5a-7968-4403-b6aa-2d83dcbff14b","Type":"ContainerDied","Data":"73c2fbac08c131884280f4fd3206a6ccedc52b3833d233546fea8191d01e34c3"}
Oct 30 21:48:31.777068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.776988    4633 scope.go:122] "RemoveContainer" containerID="dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326"
Oct 30 21:48:31.779112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.779079    4633 generic.go:356] "Generic (PLEG): container finished" podID="b905e0bb-b295-4752-b8d1-19ae58096f46" containerID="44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb" exitCode=0
Oct 30 21:48:31.779249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.779124    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"
Oct 30 21:48:31.779249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.779155    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9" event={"ID":"b905e0bb-b295-4752-b8d1-19ae58096f46","Type":"ContainerDied","Data":"44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb"}
Oct 30 21:48:31.779249 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.779200    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9" event={"ID":"b905e0bb-b295-4752-b8d1-19ae58096f46","Type":"ContainerDied","Data":"1f7ca7f3cac6db7da1b291c841724c6005a965756343d002e45b227a2746a79f"}
Oct 30 21:48:31.792524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.792501    4633 scope.go:122] "RemoveContainer" containerID="dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326"
Oct 30 21:48:31.805638 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:31.805602    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326\": container with ID starting with dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326 not found: ID does not exist" containerID="dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326"
Oct 30 21:48:31.805953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.805917    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326"} err="failed to get container status \"dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326\": rpc error: code = NotFound desc = could not find container \"dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326\": container with ID starting with dc3d67c959e0b629224d347b02a4d100bc0c6c10c7fc020a0ab735914246c326 not found: ID does not exist"
Oct 30 21:48:31.805953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.805950    4633 scope.go:122] "RemoveContainer" containerID="44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb"
Oct 30 21:48:31.832924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.832893    4633 scope.go:122] "RemoveContainer" containerID="44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb"
Oct 30 21:48:31.833599 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:31.833504    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb\": container with ID starting with 44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb not found: ID does not exist" containerID="44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb"
Oct 30 21:48:31.833599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.833545    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb"} err="failed to get container status \"44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb\": rpc error: code = NotFound desc = could not find container \"44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb\": container with ID starting with 44f5139a382d20f052a5cea219efdd6d57774e1f6b09c43ea8e701b6d24a2ebb not found: ID does not exist"
Oct 30 21:48:31.841535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.841496    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"]
Oct 30 21:48:31.848066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.848038    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b30a42-59bb49b88c-lxzf9"]
Oct 30 21:48:31.864867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.864822    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"]
Oct 30 21:48:31.889062 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:31.889024    4633 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e11b5a_7968_4403_b6aa_2d83dcbff14b.slice/crio-73c2fbac08c131884280f4fd3206a6ccedc52b3833d233546fea8191d01e34c3\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb905e0bb_b295_4752_b8d1_19ae58096f46.slice/crio-1f7ca7f3cac6db7da1b291c841724c6005a965756343d002e45b227a2746a79f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb905e0bb_b295_4752_b8d1_19ae58096f46.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e11b5a_7968_4403_b6aa_2d83dcbff14b.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:48:31.889240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:31.889072    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b30a42-84bcb96469-wxw5d"]
Oct 30 21:48:32.115255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:32.115124    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"]
Oct 30 21:48:32.115473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:32.115416    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy" containerID="cri-o://ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05" gracePeriod=30
Oct 30 21:48:33.793042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:33.793008    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e11b5a-7968-4403-b6aa-2d83dcbff14b" path="/var/lib/kubelet/pods/05e11b5a-7968-4403-b6aa-2d83dcbff14b/volumes"
Oct 30 21:48:33.793523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:33.793508    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b905e0bb-b295-4752-b8d1-19ae58096f46" path="/var/lib/kubelet/pods/b905e0bb-b295-4752-b8d1-19ae58096f46/volumes"
Oct 30 21:48:35.686631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:35.685324    4633 scope.go:122] "RemoveContainer" containerID="7f43de400f408700618aea3bbe7a3e2f3a347c0922120776750e5888f8cc2830"
Oct 30 21:48:35.699435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:35.699410    4633 scope.go:122] "RemoveContainer" containerID="9bd3c85e64b1f1427c2f821f122b0c3c0ab60e15646ed7c7f903975e892348a8"
Oct 30 21:48:37.416111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.416084    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:48:37.501531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.501490    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-token\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-token\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.501838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.501562    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-credential-socket\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.501838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.501657    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c175ae88-19d0-4c92-8f35-649ac11138ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175ae88-19d0-4c92-8f35-649ac11138ac-istiod-ca-cert\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.501838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.501776    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-data\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.501838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.501802    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-certs\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.502118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.501872    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-podinfo\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.502118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.502051    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-socket\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.502254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.502155    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-envoy\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.502254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.502226    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-kube-api-access-7xrcg\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-kube-api-access-7xrcg\") pod \"c175ae88-19d0-4c92-8f35-649ac11138ac\" (UID: \"c175ae88-19d0-4c92-8f35-649ac11138ac\") "
Oct 30 21:48:37.509250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.507883    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c175ae88-19d0-4c92-8f35-649ac11138ac-istiod-ca-cert" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:37.509250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.508181    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-credential-socket" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.509250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.508366    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-data" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.509875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.509837    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-socket" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.520881 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.520836    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-certs" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.547921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.546911    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-envoy" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.572321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.571660    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"]
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573335    4633 cpu_manager.go:414] "containerMap: removing container" podUID="05e11b5a-7968-4403-b6aa-2d83dcbff14b" containerName="payload-processing"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573358    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e11b5a-7968-4403-b6aa-2d83dcbff14b" containerName="payload-processing"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573390    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573397    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573423    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573430    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573447    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b905e0bb-b295-4752-b8d1-19ae58096f46" containerName="payload-pre-processing"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573455    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b905e0bb-b295-4752-b8d1-19ae58096f46" containerName="payload-pre-processing"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573656    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e11b5a-7968-4403-b6aa-2d83dcbff14b" containerName="payload-processing"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573699    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573713    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerName="istio-proxy"
Oct 30 21:48:37.574765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.573723    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b905e0bb-b295-4752-b8d1-19ae58096f46" containerName="payload-pre-processing"
Oct 30 21:48:37.578201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.578158    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-podinfo" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:48:37.578523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.578479    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-token" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:37.580524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.580491    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-kube-api-access-7xrcg" pod "c175ae88-19d0-4c92-8f35-649ac11138ac" (UID: "c175ae88-19d0-4c92-8f35-649ac11138ac"). InnerVolumeSpecName "kube-api-access-7xrcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:37.580743 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.580729    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"]
Oct 30 21:48:37.580916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.580894    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.604789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.604751    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-envoy\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.604942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.604799    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-credential-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.604942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.604881    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvvj\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-kube-api-access-7rvvj\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.605067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.604943    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df70c75b-9dcd-44e9-8a6d-0195be290365-istiod-ca-cert\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.605067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.604984    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-data\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.605067 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605059    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.605237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605098    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-certs\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.605237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605116    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-podinfo\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.605432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605407    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-token\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.605600 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605580    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.605700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605606    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.605700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605622    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xrcg\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-kube-api-access-7xrcg\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.605700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605638    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.605700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605652    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.605700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605685    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c175ae88-19d0-4c92-8f35-649ac11138ac-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.605700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605699    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.606050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605713    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c175ae88-19d0-4c92-8f35-649ac11138ac-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.606050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.605726    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c175ae88-19d0-4c92-8f35-649ac11138ac-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.707291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707258    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-token\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.707463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707401    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-envoy\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.707463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707432    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-credential-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.707463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707453    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvvj\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-kube-api-access-7rvvj\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.707653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707480    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df70c75b-9dcd-44e9-8a6d-0195be290365-istiod-ca-cert\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.707653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707618    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-data\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.707860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707736    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.707860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707804    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-certs\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.707860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.707837    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-podinfo\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.708031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.708011    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-credential-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.708374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.708350    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-data\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.708486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.708420    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df70c75b-9dcd-44e9-8a6d-0195be290365-istiod-ca-cert\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.708486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.708464    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-socket\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.708612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.708508    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-certs\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.710048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.710029    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-envoy\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.710494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.710476    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-podinfo\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.719801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.719781    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvvj\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-kube-api-access-7rvvj\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.724420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.724395    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-token\") pod \"e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") " pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:37.835915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.835885    4633 generic.go:356] "Generic (PLEG): container finished" podID="c175ae88-19d0-4c92-8f35-649ac11138ac" containerID="ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05" exitCode=0
Oct 30 21:48:37.836078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.835940    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"
Oct 30 21:48:37.836078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.835972    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" event={"ID":"c175ae88-19d0-4c92-8f35-649ac11138ac","Type":"ContainerDied","Data":"ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05"}
Oct 30 21:48:37.836078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.836025    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k" event={"ID":"c175ae88-19d0-4c92-8f35-649ac11138ac","Type":"ContainerDied","Data":"e53991b922e50df4d48d2dd6805ec3b99ed12ad030555d793a51cda965ce5011"}
Oct 30 21:48:37.836078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.836047    4633 scope.go:122] "RemoveContainer" containerID="ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05"
Oct 30 21:48:37.849199 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.849179    4633 scope.go:122] "RemoveContainer" containerID="939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0"
Oct 30 21:48:37.860057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.860041    4633 scope.go:122] "RemoveContainer" containerID="ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05"
Oct 30 21:48:37.860320 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:37.860302    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05\": container with ID starting with ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05 not found: ID does not exist" containerID="ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05"
Oct 30 21:48:37.860373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.860329    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05"} err="failed to get container status \"ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05\": rpc error: code = NotFound desc = could not find container \"ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05\": container with ID starting with ece5937add3ad1d20d301d11e04a08168e112f2dda999c0e02e7fb0514f15d05 not found: ID does not exist"
Oct 30 21:48:37.860373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.860346    4633 scope.go:122] "RemoveContainer" containerID="939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0"
Oct 30 21:48:37.860565 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:37.860550    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0\": container with ID starting with 939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0 not found: ID does not exist" containerID="939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0"
Oct 30 21:48:37.860607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.860571    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0"} err="failed to get container status \"939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0\": rpc error: code = NotFound desc = could not find container \"939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0\": container with ID starting with 939a357c0a62f66f933b9dee0a11f17f5b22e27da78925c5c856c5ee70af3ea0 not found: ID does not exist"
Oct 30 21:48:37.879518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.879489    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"]
Oct 30 21:48:37.890957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.890936    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-mpf5k"]
Oct 30 21:48:37.899204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:37.899178    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:38.198810 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:48:38.198780    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf70c75b_9dcd_44e9_8a6d_0195be290365.slice/crio-3ba4d9cff7d0caa208f6459b68aca36c9e3cef9c15b5ba00568196d064344450 WatchSource:0}: Error finding container 3ba4d9cff7d0caa208f6459b68aca36c9e3cef9c15b5ba00568196d064344450: Status 404 returned error can't find the container with id 3ba4d9cff7d0caa208f6459b68aca36c9e3cef9c15b5ba00568196d064344450
Oct 30 21:48:38.199276 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:38.199254    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"]
Oct 30 21:48:38.202465 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:38.202430    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:38.202548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:38.202528    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:38.202610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:38.202558    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:38.847449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:38.847412    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" event={"ID":"df70c75b-9dcd-44e9-8a6d-0195be290365","Type":"ContainerStarted","Data":"e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399"}
Oct 30 21:48:38.847974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:38.847474    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" event={"ID":"df70c75b-9dcd-44e9-8a6d-0195be290365","Type":"ContainerStarted","Data":"3ba4d9cff7d0caa208f6459b68aca36c9e3cef9c15b5ba00568196d064344450"}
Oct 30 21:48:38.870283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:38.870229    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podStartSLOduration=12.870214298 podStartE2EDuration="12.870214298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:38.868498122 +0000 UTC m=+2827.218225531" watchObservedRunningTime="2026-10-30 21:48:38.870214298 +0000 UTC m=+2827.219941769"
Oct 30 21:48:38.899997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:38.899958    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:39.844693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:39.843899    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c175ae88-19d0-4c92-8f35-649ac11138ac" path="/var/lib/kubelet/pods/c175ae88-19d0-4c92-8f35-649ac11138ac/volumes"
Oct 30 21:48:39.929697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:39.928803    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:48:39.929697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:39.928865    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 21:48:40.498781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:40.498748    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:48:40.904704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:40.903875    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:48:40.904704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:40.903933    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 21:48:41.923701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:41.914809    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:48:41.923701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:41.914876    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 21:48:42.909924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:42.909873    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:48:42.910128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:42.909952    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 21:48:43.899968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:43.899923    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:48:43.900397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:43.899991    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 21:48:44.900066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:44.900034    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:48:44.900507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:44.900084    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 21:48:45.899913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:45.899869    4633 patch_prober.go:29] interesting pod/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 30 21:48:45.900107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:45.899935    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 30 21:48:46.905933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:46.905899    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:46.906474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:46.906453    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:46.907250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:46.907227    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:48:49.360023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.359988    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"]
Oct 30 21:48:49.366329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.366301    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.389463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.389431    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cf51d4\"/\"openshift-service-ca.crt\""
Oct 30 21:48:49.389693 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.389661    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cf51d4\"/\"default-dockercfg-wxnb9\""
Oct 30 21:48:49.390383 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.390360    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cf51d4\"/\"test-model-cf51d4-kserve-self-signed-certs\""
Oct 30 21:48:49.392389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.392367    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cf51d4\"/\"kube-root-ca.crt\""
Oct 30 21:48:49.392656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.392632    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8023d07-bf60-418f-b26c-1ef210b5f032-tls-certs\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.392902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.392880    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pf8x\" (UniqueName: \"kubernetes.io/projected/f8023d07-bf60-418f-b26c-1ef210b5f032-kube-api-access-5pf8x\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.393019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.392929    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-dshm\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.393115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.393101    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-model-cache\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.393279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.393263    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-tmp-dir\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.393335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.393319    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-home\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.409171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.409132    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"]
Oct 30 21:48:49.494634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.494600    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-tmp-dir\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.494892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.494700    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-home\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.494892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.494785    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8023d07-bf60-418f-b26c-1ef210b5f032-tls-certs\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.494892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.494834    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf8x\" (UniqueName: \"kubernetes.io/projected/f8023d07-bf60-418f-b26c-1ef210b5f032-kube-api-access-5pf8x\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.494892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.494876    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-dshm\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.495150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.495037    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-model-cache\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.495270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.495241    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-tmp-dir\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.495474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.495447    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-model-cache\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.495938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.495916    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-home\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.498149 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.498121    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-dshm\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.498306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.498288    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8023d07-bf60-418f-b26c-1ef210b5f032-tls-certs\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.507484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.507459    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf8x\" (UniqueName: \"kubernetes.io/projected/f8023d07-bf60-418f-b26c-1ef210b5f032-kube-api-access-5pf8x\") pod \"test-model-cf51d4-kserve-8df4bc6d9-h9f4f\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") " pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:49.690750 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:49.686012    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:50.235795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:50.235752    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"]
Oct 30 21:48:50.830356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:50.830313    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"]
Oct 30 21:48:50.830916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:50.830618    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp" podUID="408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" containerName="maas-api" containerID="cri-o://40ac13810f42e031762d8913e597be1fe37642496a17f43d6b691532ffdeef22" gracePeriod=30
Oct 30 21:48:51.477543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.474050    4633 generic.go:356] "Generic (PLEG): container finished" podID="408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" containerID="40ac13810f42e031762d8913e597be1fe37642496a17f43d6b691532ffdeef22" exitCode=0
Oct 30 21:48:51.477543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.474218    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp" event={"ID":"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0","Type":"ContainerDied","Data":"40ac13810f42e031762d8913e597be1fe37642496a17f43d6b691532ffdeef22"}
Oct 30 21:48:51.522689 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.522637    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f" event={"ID":"f8023d07-bf60-418f-b26c-1ef210b5f032","Type":"ContainerStarted","Data":"a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605"}
Oct 30 21:48:51.522948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.522928    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f" event={"ID":"f8023d07-bf60-418f-b26c-1ef210b5f032","Type":"ContainerStarted","Data":"663133f5efe307b09a088c6f4b49f421e50bf7314c8ff1f9ccb2f8d7900d2b06"}
Oct 30 21:48:51.548565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.548535    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:48:51.549895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.549852    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f" podStartSLOduration=2.5498380899999997 podStartE2EDuration="2.54983809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:51.545564345 +0000 UTC m=+2839.895291742" watchObservedRunningTime="2026-10-30 21:48:51.54983809 +0000 UTC m=+2839.899565486"
Oct 30 21:48:51.620964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.620930    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-metrics-certs\") pod \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") "
Oct 30 21:48:51.621112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.621071    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-openshift-service-ca\") pod \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") "
Oct 30 21:48:51.621312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.621295    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-maas-api-tls\") pod \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") "
Oct 30 21:48:51.621389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.621360    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-kube-api-access-zkvkf\" (UniqueName: \"kubernetes.io/projected/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-kube-api-access-zkvkf\") pod \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\" (UID: \"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0\") "
Oct 30 21:48:51.621535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.621403    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-openshift-service-ca" pod "408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" (UID: "408f0dbe-3026-47c0-bbfa-8c8d6845a7d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:51.622160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.622135    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:51.623502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.623470    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-metrics-certs" pod "408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" (UID: "408f0dbe-3026-47c0-bbfa-8c8d6845a7d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:51.623605 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.623534    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-kube-api-access-zkvkf" pod "408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" (UID: "408f0dbe-3026-47c0-bbfa-8c8d6845a7d0"). InnerVolumeSpecName "kube-api-access-zkvkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:51.623655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.623612    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-maas-api-tls" pod "408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" (UID: "408f0dbe-3026-47c0-bbfa-8c8d6845a7d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:51.706695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.688174    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:51.760060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.759958    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:51.760060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.759995    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:51.760060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.760009    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkvkf\" (UniqueName: \"kubernetes.io/projected/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0-kube-api-access-zkvkf\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:51.773300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.773264    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"]
Oct 30 21:48:51.774552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.774534    4633 cpu_manager.go:414] "containerMap: removing container" podUID="408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" containerName="maas-api"
Oct 30 21:48:51.774630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.774555    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" containerName="maas-api"
Oct 30 21:48:51.774815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.774803    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" containerName="maas-api"
Oct 30 21:48:51.779589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.779525    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.799942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.799917    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"]
Oct 30 21:48:51.962829 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.962780    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.963310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.963064    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.963310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.963181    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.963310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.963214    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.963310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.963277    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.963503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.963375    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.963503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.963420    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.963639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.963618    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmm84\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-kube-api-access-vmm84\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:51.963793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:51.963769    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521156d-a012-4fe7-8873-a30d24fa3bdd-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.065515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.065420    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmm84\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-kube-api-access-vmm84\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.065723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.065540    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521156d-a012-4fe7-8873-a30d24fa3bdd-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.065723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.065657    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.065914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.065808    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.065914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.065880    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.065914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.065904    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.066106 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.065941    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.066106 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.066008    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.066106 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.066046    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.066281 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.066174    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.066341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.066320    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.066425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.066395    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.066496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.066400    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521156d-a012-4fe7-8873-a30d24fa3bdd-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.066905 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.066882    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.068885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.068862    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.069229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.069209    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.083612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.083589    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.085612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.085588    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmm84\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-kube-api-access-vmm84\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.099204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.099186    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:52.410507 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:48:52.410472    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda521156d_a012_4fe7_8873_a30d24fa3bdd.slice/crio-b880ca41cfd6b34083297c62fb3ce75730f3140cb5dc5f93334f57fb9480bca4 WatchSource:0}: Error finding container b880ca41cfd6b34083297c62fb3ce75730f3140cb5dc5f93334f57fb9480bca4: Status 404 returned error can't find the container with id b880ca41cfd6b34083297c62fb3ce75730f3140cb5dc5f93334f57fb9480bca4
Oct 30 21:48:52.410878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.410857    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"]
Oct 30 21:48:52.413968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.413933    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:52.414068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.414005    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:52.414068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.414034    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:48:52.534271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.533964    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" event={"ID":"a521156d-a012-4fe7-8873-a30d24fa3bdd","Type":"ContainerStarted","Data":"1a7effacddc4f3cc73689015d0f788f5170118c25c7700fff7a474cba8bc19ea"}
Oct 30 21:48:52.534271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.534044    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" event={"ID":"a521156d-a012-4fe7-8873-a30d24fa3bdd","Type":"ContainerStarted","Data":"b880ca41cfd6b34083297c62fb3ce75730f3140cb5dc5f93334f57fb9480bca4"}
Oct 30 21:48:52.535895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.535873    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"
Oct 30 21:48:52.536039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.535906    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp" event={"ID":"408f0dbe-3026-47c0-bbfa-8c8d6845a7d0","Type":"ContainerDied","Data":"0ba3ec7e1c564152cc501c3eeb361c0f584de5c45d46629160808c2891a14ec6"}
Oct 30 21:48:52.536039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.535968    4633 scope.go:122] "RemoveContainer" containerID="40ac13810f42e031762d8913e597be1fe37642496a17f43d6b691532ffdeef22"
Oct 30 21:48:52.610692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.608885    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podStartSLOduration=1.608860939 podStartE2EDuration="1.608860939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:52.601112335 +0000 UTC m=+2840.950839731" watchObservedRunningTime="2026-10-30 21:48:52.608860939 +0000 UTC m=+2840.958588336"
Oct 30 21:48:52.640615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.640578    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"]
Oct 30 21:48:52.661376 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:52.661293    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-128d2f-69bdbd494d-f8xsp"]
Oct 30 21:48:53.100205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:53.100108    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:48:53.102135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:53.102106    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:48:53.102264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:53.102164    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 21:48:53.861389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:53.860452    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408f0dbe-3026-47c0-bbfa-8c8d6845a7d0" path="/var/lib/kubelet/pods/408f0dbe-3026-47c0-bbfa-8c8d6845a7d0/volumes"
Oct 30 21:48:54.100262 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:54.100226    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:48:54.100721 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:54.100285    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 21:48:55.052398 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:48:55.052357    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:48:55.099941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:55.099904    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:48:55.100123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:55.099955    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 21:48:56.100434 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:56.100398    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:48:56.100853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:56.100465    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 21:48:57.100059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:57.100024    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:48:57.100250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:57.100086    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 21:48:58.100052 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:58.100013    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:48:58.100453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:58.100068    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 21:48:59.100440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:59.100406    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 21:48:59.100974 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:59.100469    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 21:48:59.687193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:59.687163    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:59.816594 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:59.816559    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:48:59.824941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:59.824916    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"]
Oct 30 21:48:59.825284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:48:59.825263    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy" containerID="cri-o://42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886" gracePeriod=30
Oct 30 21:49:00.328377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.328346    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:49:00.330021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.330001    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:49:00.330718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.330653    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:49:00.546236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.546195    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"]
Oct 30 21:49:00.551730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.551705    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.565566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.565540    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"]
Oct 30 21:49:00.615462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.615376    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:49:00.615644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.615590    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:49:00.690695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.690303    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.690695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.690474    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.705027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.701032    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qmp\" (UniqueName: \"kubernetes.io/projected/380d1dcd-e949-4e57-97e1-49924f762d7d-kube-api-access-q9qmp\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.722059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.722027    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:49:00.802843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.802796    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.803181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.803161    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qmp\" (UniqueName: \"kubernetes.io/projected/380d1dcd-e949-4e57-97e1-49924f762d7d-kube-api-access-q9qmp\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.803344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.803328    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.803599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.803575    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.806558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.806529    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.812619 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.812587    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qmp\" (UniqueName: \"kubernetes.io/projected/380d1dcd-e949-4e57-97e1-49924f762d7d-kube-api-access-q9qmp\") pod \"maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:00.871334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:00.871254    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:01.169871 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:49:01.169829    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380d1dcd_e949_4e57_97e1_49924f762d7d.slice/crio-3375a5ad07efacd1f746cee1c773e28d25ca648ebdbbf760dcbd4560185aa3a6 WatchSource:0}: Error finding container 3375a5ad07efacd1f746cee1c773e28d25ca648ebdbbf760dcbd4560185aa3a6: Status 404 returned error can't find the container with id 3375a5ad07efacd1f746cee1c773e28d25ca648ebdbbf760dcbd4560185aa3a6
Oct 30 21:49:01.170591 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:01.170567    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"]
Oct 30 21:49:01.768718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:01.768641    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs" event={"ID":"380d1dcd-e949-4e57-97e1-49924f762d7d","Type":"ContainerStarted","Data":"849dd05bd0563df394be52032da0815bd1557af6b2d49178d7acfffe9e8d4c8d"}
Oct 30 21:49:01.769638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:01.768758    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs" event={"ID":"380d1dcd-e949-4e57-97e1-49924f762d7d","Type":"ContainerStarted","Data":"3375a5ad07efacd1f746cee1c773e28d25ca648ebdbbf760dcbd4560185aa3a6"}
Oct 30 21:49:01.787108 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:01.787015    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs" podStartSLOduration=1.7870021390000002 podStartE2EDuration="1.787002139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:01.784797114 +0000 UTC m=+2850.134524511" watchObservedRunningTime="2026-10-30 21:49:01.787002139 +0000 UTC m=+2850.136729536"
Oct 30 21:49:02.549316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.549223    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"]
Oct 30 21:49:02.554476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.554446    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.564717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.561521    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-87ae68\"/\"kube-root-ca.crt\""
Oct 30 21:49:02.564717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.561841    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-87ae68\"/\"test-model-87ae68-kserve-self-signed-certs\""
Oct 30 21:49:02.564717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.562337    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-87ae68\"/\"openshift-service-ca.crt\""
Oct 30 21:49:02.564717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.562419    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-87ae68\"/\"default-dockercfg-x6rlc\""
Oct 30 21:49:02.565936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.565914    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"]
Oct 30 21:49:02.647620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.647587    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tmp-dir\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.647910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.647627    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-home\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.647910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.647775    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tls-certs\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.647910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.647830    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-dshm\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.648118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.648100    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbn5\" (UniqueName: \"kubernetes.io/projected/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-kube-api-access-wwbn5\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.648161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.648137    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-model-cache\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.776038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.775993    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbn5\" (UniqueName: \"kubernetes.io/projected/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-kube-api-access-wwbn5\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.776661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.776630    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-model-cache\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.776953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.776931    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tmp-dir\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.777091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.777073    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-home\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.777312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.777295    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tls-certs\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.777473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.777458    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-dshm\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.781702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.781236    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tmp-dir\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.781900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.781853    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-model-cache\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.789698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.782293    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-home\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.854905 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.852314    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbn5\" (UniqueName: \"kubernetes.io/projected/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-kube-api-access-wwbn5\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.878695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.869051    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-dshm\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.878695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.869646    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tls-certs\") pod \"test-model-87ae68-kserve-8bdd945bf-z4lgh\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") " pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.885944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.885912    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:02.907566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.887931    4633 generic.go:356] "Generic (PLEG): container finished" podID="380d1dcd-e949-4e57-97e1-49924f762d7d" containerID="849dd05bd0563df394be52032da0815bd1557af6b2d49178d7acfffe9e8d4c8d" exitCode=0
Oct 30 21:49:02.908082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:02.887959    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs" event={"ID":"380d1dcd-e949-4e57-97e1-49924f762d7d","Type":"ContainerDied","Data":"849dd05bd0563df394be52032da0815bd1557af6b2d49178d7acfffe9e8d4c8d"}
Oct 30 21:49:03.334353 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:49:03.334309    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1aa404d_1c9e_47ec_b67d_e3f8124a17eb.slice/crio-420b71bf6c4f9063c284006acfc42990627b1ef4cf36abefc68ec33903cdd06a WatchSource:0}: Error finding container 420b71bf6c4f9063c284006acfc42990627b1ef4cf36abefc68ec33903cdd06a: Status 404 returned error can't find the container with id 420b71bf6c4f9063c284006acfc42990627b1ef4cf36abefc68ec33903cdd06a
Oct 30 21:49:03.335065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:03.335038    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"]
Oct 30 21:49:03.918493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:03.918455    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh" event={"ID":"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb","Type":"ContainerStarted","Data":"21e4bb58c001f6a69869d6993617587cd38be488651a5f9783e05c0afa505638"}
Oct 30 21:49:03.919045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:03.918506    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh" event={"ID":"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb","Type":"ContainerStarted","Data":"420b71bf6c4f9063c284006acfc42990627b1ef4cf36abefc68ec33903cdd06a"}
Oct 30 21:49:03.942441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:03.942389    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh" podStartSLOduration=1.942375857 podStartE2EDuration="1.942375857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:03.935899737 +0000 UTC m=+2852.285627135" watchObservedRunningTime="2026-10-30 21:49:03.942375857 +0000 UTC m=+2852.292103253"
Oct 30 21:49:04.194870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.194844    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:04.272694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.258297    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380d1dcd-e949-4e57-97e1-49924f762d7d-kube-api-access-q9qmp\" (UniqueName: \"kubernetes.io/projected/380d1dcd-e949-4e57-97e1-49924f762d7d-kube-api-access-q9qmp\") pod \"380d1dcd-e949-4e57-97e1-49924f762d7d\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") "
Oct 30 21:49:04.272694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.258548    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-openshift-ca\") pod \"380d1dcd-e949-4e57-97e1-49924f762d7d\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") "
Oct 30 21:49:04.272694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.258740    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-xks-ca\") pod \"380d1dcd-e949-4e57-97e1-49924f762d7d\" (UID: \"380d1dcd-e949-4e57-97e1-49924f762d7d\") "
Oct 30 21:49:04.303246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.303210    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-openshift-ca" pod "380d1dcd-e949-4e57-97e1-49924f762d7d" (UID: "380d1dcd-e949-4e57-97e1-49924f762d7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:04.330211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.322941    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-xks-ca" pod "380d1dcd-e949-4e57-97e1-49924f762d7d" (UID: "380d1dcd-e949-4e57-97e1-49924f762d7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:04.330211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.323913    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380d1dcd-e949-4e57-97e1-49924f762d7d-kube-api-access-q9qmp" pod "380d1dcd-e949-4e57-97e1-49924f762d7d" (UID: "380d1dcd-e949-4e57-97e1-49924f762d7d"). InnerVolumeSpecName "kube-api-access-q9qmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:04.360885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.360855    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9qmp\" (UniqueName: \"kubernetes.io/projected/380d1dcd-e949-4e57-97e1-49924f762d7d-kube-api-access-q9qmp\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:04.360885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.360883    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-openshift-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:04.361200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.360893    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380d1dcd-e949-4e57-97e1-49924f762d7d-maas-api-xks-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:04.886755 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.886723    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:04.930963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.929773    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"
Oct 30 21:49:04.930963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.929979    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs" event={"ID":"380d1dcd-e949-4e57-97e1-49924f762d7d","Type":"ContainerDied","Data":"3375a5ad07efacd1f746cee1c773e28d25ca648ebdbbf760dcbd4560185aa3a6"}
Oct 30 21:49:04.930963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:04.930050    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3375a5ad07efacd1f746cee1c773e28d25ca648ebdbbf760dcbd4560185aa3a6"
Oct 30 21:49:05.276387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.276361    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:49:05.456787 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.456749    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-token\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-token\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.456955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.456889    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-podinfo\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.456955 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.456927    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-certs\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.457065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.456987    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-kube-api-access-v4pwv\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-kube-api-access-v4pwv\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.457109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457080    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-data\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.457272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457238    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-certs" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:05.457427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457281    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-socket\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.457427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457314    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-envoy\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.457427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457333    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1092b6c6-b82a-4c8d-bf08-af328c036781-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1092b6c6-b82a-4c8d-bf08-af328c036781-istiod-ca-cert\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.457615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457456    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-credential-socket\") pod \"1092b6c6-b82a-4c8d-bf08-af328c036781\" (UID: \"1092b6c6-b82a-4c8d-bf08-af328c036781\") "
Oct 30 21:49:05.457615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457459    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-data" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:05.457615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457542    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-socket" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:05.457856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.457831    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-credential-socket" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:05.458050 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.458024    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1092b6c6-b82a-4c8d-bf08-af328c036781-istiod-ca-cert" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:05.458323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.458304    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:05.458415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.458327    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:05.458415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.458337    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:05.458415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.458346    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1092b6c6-b82a-4c8d-bf08-af328c036781-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:05.458415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.458355    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:05.459663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.459642    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-envoy" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:05.459663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.459703    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-podinfo" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:49:05.459944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.459925    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-kube-api-access-v4pwv" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "kube-api-access-v4pwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:05.460014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.459958    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-token" pod "1092b6c6-b82a-4c8d-bf08-af328c036781" (UID: "1092b6c6-b82a-4c8d-bf08-af328c036781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:05.560349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.560260    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:05.560349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.560299    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:05.560349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.560314    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1092b6c6-b82a-4c8d-bf08-af328c036781-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:05.560349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:05.560328    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4pwv\" (UniqueName: \"kubernetes.io/projected/1092b6c6-b82a-4c8d-bf08-af328c036781-kube-api-access-v4pwv\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:06.008089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.008057    4633 generic.go:356] "Generic (PLEG): container finished" podID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerID="42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886" exitCode=0
Oct 30 21:49:06.008732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.008710    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"
Oct 30 21:49:06.008950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.008663    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" event={"ID":"1092b6c6-b82a-4c8d-bf08-af328c036781","Type":"ContainerDied","Data":"42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886"}
Oct 30 21:49:06.009032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.008997    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54" event={"ID":"1092b6c6-b82a-4c8d-bf08-af328c036781","Type":"ContainerDied","Data":"47e960fe24edfa344b331eec4fa956cd98ac9d397b46aa14785a13efa009682c"}
Oct 30 21:49:06.009087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.009035    4633 scope.go:122] "RemoveContainer" containerID="42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886"
Oct 30 21:49:06.023626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.023609    4633 scope.go:122] "RemoveContainer" containerID="730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a"
Oct 30 21:49:06.038639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.038613    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"]
Oct 30 21:49:06.049592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.043323    4633 scope.go:122] "RemoveContainer" containerID="42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886"
Oct 30 21:49:06.049592 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:06.043804    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886\": container with ID starting with 42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886 not found: ID does not exist" containerID="42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886"
Oct 30 21:49:06.049592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.043843    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886"} err="failed to get container status \"42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886\": rpc error: code = NotFound desc = could not find container \"42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886\": container with ID starting with 42576f9904493da2e200c4348800669a9723d2fe5661c2a5285815741dd55886 not found: ID does not exist"
Oct 30 21:49:06.049592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.043868    4633 scope.go:122] "RemoveContainer" containerID="730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a"
Oct 30 21:49:06.049592 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:06.044813    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a\": container with ID starting with 730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a not found: ID does not exist" containerID="730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a"
Oct 30 21:49:06.049592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.044843    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a"} err="failed to get container status \"730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a\": rpc error: code = NotFound desc = could not find container \"730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a\": container with ID starting with 730a2fee953536d7db59816dac3de075cb4a15d3d0ec02cba2f0e2abc0bc4f2a not found: ID does not exist"
Oct 30 21:49:06.054227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:06.054204    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-128d2f-openshift-default-59c97659ff-sss54"]
Oct 30 21:49:07.765875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.765842    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"]
Oct 30 21:49:07.766691 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766660    4633 cpu_manager.go:414] "containerMap: removing container" podUID="380d1dcd-e949-4e57-97e1-49924f762d7d" containerName="revoke-keys"
Oct 30 21:49:07.766751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766695    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380d1dcd-e949-4e57-97e1-49924f762d7d" containerName="revoke-keys"
Oct 30 21:49:07.766751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766725    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy"
Oct 30 21:49:07.766751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766731    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy"
Oct 30 21:49:07.766751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766742    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy"
Oct 30 21:49:07.766751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766747    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy"
Oct 30 21:49:07.766916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766884    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="380d1dcd-e949-4e57-97e1-49924f762d7d" containerName="revoke-keys"
Oct 30 21:49:07.766916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766898    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy"
Oct 30 21:49:07.766916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.766904    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" containerName="istio-proxy"
Oct 30 21:49:07.772423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.772403    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:07.779289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.779174    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"payload-processing-external-model-plugins-e2e-extmod-a-fb2b58\""
Oct 30 21:49:07.779289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.779186    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"payload-processing-external-model-e2e--b5c8139c-dockercfg-b6q5w\""
Oct 30 21:49:07.779289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.779181    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"openshift-service-ca.crt\""
Oct 30 21:49:07.779289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.779224    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"agc-cred-98afef9e5c37\""
Oct 30 21:49:07.779289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.779265    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"routing-overlay\""
Oct 30 21:49:07.779289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.779284    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fb2b58\"/\"kube-root-ca.crt\""
Oct 30 21:49:07.804193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.804162    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1092b6c6-b82a-4c8d-bf08-af328c036781" path="/var/lib/kubelet/pods/1092b6c6-b82a-4c8d-bf08-af328c036781/volumes"
Oct 30 21:49:07.804732 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.804713    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"]
Oct 30 21:49:07.964696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.954450    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8249\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-kube-api-access-r8249\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.008771 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:07.985002    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.100095 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.100013    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.104957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.104919    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.206763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.206729    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.206979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.206837    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.206979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.206898    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.207122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.207107    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8249\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-kube-api-access-r8249\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.207586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.207557    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.207692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.207588    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.209452 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.209435    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.217657 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.217634    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8249\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-kube-api-access-r8249\") pod \"payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") " pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.368825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.368731    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"]
Oct 30 21:49:08.404282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.404245    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:08.499700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.498984    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:08.525868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.511596    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"]
Oct 30 21:49:08.665372 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.656714    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"routing-overlay\""
Oct 30 21:49:08.965927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.675134    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"openshift-service-ca.crt\""
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.675219    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"payload-processing-external-model-e2e--8e1d66cd-dockercfg-d8wks\""
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.676142    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"kube-root-ca.crt\""
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.676179    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"agc-cred-00b228f0efa6\""
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.676218    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea\""
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:08.679173    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.221759    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.221859    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.221935    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj58c\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-kube-api-access-qj58c\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.231038    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"]
Oct 30 21:49:09.258983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.231549    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt" podUID="18a7653d-8782-4705-b398-e89d2d9d1fb2" containerName="sim" containerID="cri-o://ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f" gracePeriod=30
Oct 30 21:49:09.727716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.682644    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj58c\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-kube-api-access-qj58c\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.738224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.719815    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj58c\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-kube-api-access-qj58c\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.738443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.738417    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.743705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.738766    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.743705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.738846    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.743705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.740199    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.743705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.740856    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.821214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.821063    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:09.916464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:09.916429    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:10.045422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.045387    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:49:10.091378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.091346    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"]
Oct 30 21:49:10.265444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.231737    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a7653d-8782-4705-b398-e89d2d9d1fb2-kube-api-access-hvggz\" (UniqueName: \"kubernetes.io/projected/18a7653d-8782-4705-b398-e89d2d9d1fb2-kube-api-access-hvggz\") pod \"18a7653d-8782-4705-b398-e89d2d9d1fb2\" (UID: \"18a7653d-8782-4705-b398-e89d2d9d1fb2\") "
Oct 30 21:49:10.265444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.231823    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a7653d-8782-4705-b398-e89d2d9d1fb2-tls\" (UniqueName: \"kubernetes.io/secret/18a7653d-8782-4705-b398-e89d2d9d1fb2-tls\") pod \"18a7653d-8782-4705-b398-e89d2d9d1fb2\" (UID: \"18a7653d-8782-4705-b398-e89d2d9d1fb2\") "
Oct 30 21:49:10.308098 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.308061    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a7653d-8782-4705-b398-e89d2d9d1fb2-tls" pod "18a7653d-8782-4705-b398-e89d2d9d1fb2" (UID: "18a7653d-8782-4705-b398-e89d2d9d1fb2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:10.315709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.308460    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a7653d-8782-4705-b398-e89d2d9d1fb2-kube-api-access-hvggz" pod "18a7653d-8782-4705-b398-e89d2d9d1fb2" (UID: "18a7653d-8782-4705-b398-e89d2d9d1fb2"). InnerVolumeSpecName "kube-api-access-hvggz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:10.402560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.401451    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvggz\" (UniqueName: \"kubernetes.io/projected/18a7653d-8782-4705-b398-e89d2d9d1fb2-kube-api-access-hvggz\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:10.403808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.403783    4633 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/18a7653d-8782-4705-b398-e89d2d9d1fb2-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:10.446764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.446729    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"]
Oct 30 21:49:10.694272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.694236    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" event={"ID":"620e8c62-3ca5-4bbe-a28c-dda117ece6d2","Type":"ContainerStarted","Data":"284dca055e2b6e1f812dee05ca740e7df990a82fe3ed523aa1ba4ec40211d6a3"}
Oct 30 21:49:10.694491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.694302    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" event={"ID":"620e8c62-3ca5-4bbe-a28c-dda117ece6d2","Type":"ContainerStarted","Data":"82b5287e9936d33eec2b3dbb38e0ecac5f3ec16e2f28c3cb6bfcc93fd4be3530"}
Oct 30 21:49:10.694617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.694582    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:10.698522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.698494    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" event={"ID":"c2a459c8-1a31-402c-9927-4bff0d3d4645","Type":"ContainerStarted","Data":"c6c33b6f0867781519e7a39a358c21cb8d8b1358bdc5e837cae892dbadb8ba40"}
Oct 30 21:49:10.698643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.698554    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:10.698643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.698577    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" event={"ID":"c2a459c8-1a31-402c-9927-4bff0d3d4645","Type":"ContainerStarted","Data":"b431b7283183920c772b36c6d6841f2d5203821d10028da8783fa652d0298150"}
Oct 30 21:49:10.700634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.700611    4633 generic.go:356] "Generic (PLEG): container finished" podID="18a7653d-8782-4705-b398-e89d2d9d1fb2" containerID="ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f" exitCode=0
Oct 30 21:49:10.700873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.700652    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"
Oct 30 21:49:10.700999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.700660    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt" event={"ID":"18a7653d-8782-4705-b398-e89d2d9d1fb2","Type":"ContainerDied","Data":"ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f"}
Oct 30 21:49:10.700999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.700967    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt" event={"ID":"18a7653d-8782-4705-b398-e89d2d9d1fb2","Type":"ContainerDied","Data":"0c4bbf11f165ad927afd05c02696cb4457015d73e74e46f570dd3db7e66c42bc"}
Oct 30 21:49:10.700999 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.700991    4633 scope.go:122] "RemoveContainer" containerID="ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f"
Oct 30 21:49:10.715888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.715870    4633 scope.go:122] "RemoveContainer" containerID="ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f"
Oct 30 21:49:10.716185 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:10.716163    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f\": container with ID starting with ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f not found: ID does not exist" containerID="ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f"
Oct 30 21:49:10.716280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.716193    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f"} err="failed to get container status \"ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f\": rpc error: code = NotFound desc = could not find container \"ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f\": container with ID starting with ccb888fd0f2e8ddb6c9825bfcd6ad953d644ee62da91d558adbc21715073da6f not found: ID does not exist"
Oct 30 21:49:10.731435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.731389    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" podStartSLOduration=3.731375916 podStartE2EDuration="3.731375916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:10.728470219 +0000 UTC m=+2859.078197615" watchObservedRunningTime="2026-10-30 21:49:10.731375916 +0000 UTC m=+2859.081103312"
Oct 30 21:49:10.752866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.752834    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"]
Oct 30 21:49:10.758181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.758155    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-0230a6/e2e-usage-sim-e580cf-54cdc9877d-mqngt"]
Oct 30 21:49:10.781871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.781810    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" podStartSLOduration=2.78179187 podStartE2EDuration="2.78179187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:10.776287575 +0000 UTC m=+2859.126014973" watchObservedRunningTime="2026-10-30 21:49:10.78179187 +0000 UTC m=+2859.131519268"
Oct 30 21:49:10.797044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:10.797014    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"]
Oct 30 21:49:10.811872 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:10.811797    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fb2b58/agc-cred-98afef9e5c37: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:10.812077 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:10.812063    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:10.813301 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:10.813278    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials podName:620e8c62-3ca5-4bbe-a28c-dda117ece6d2 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:11.313256081 +0000 UTC m=+2859.662983472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2") : secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:10.813301 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:10.812550    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0ea1ea/agc-cred-00b228f0efa6: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:10.813530 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:10.813320    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:10.813530 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:10.813378    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:11.313357675 +0000 UTC m=+2859.663085065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:11.181237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:11.166013    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"]
Oct 30 21:49:11.365695 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:11.365144    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0ea1ea/agc-cred-00b228f0efa6: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:11.365695 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:11.365177    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:11.365695 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:11.365245    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.365228945 +0000 UTC m=+2860.714956319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:11.365695 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:11.365636    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fb2b58/agc-cred-98afef9e5c37: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:11.365695 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:11.365654    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:11.366130 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:11.365725    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials podName:620e8c62-3ca5-4bbe-a28c-dda117ece6d2 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.36570799 +0000 UTC m=+2860.715435366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2") : secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:11.939934 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:11.939897    4633 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" secret="" err="secret \"payload-processing-external-model-e2e--8e1d66cd-dockercfg-d8wks\" not found"
Oct 30 21:49:11.954486 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:11.953434    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a7653d-8782-4705-b398-e89d2d9d1fb2" path="/var/lib/kubelet/pods/18a7653d-8782-4705-b398-e89d2d9d1fb2/volumes"
Oct 30 21:49:12.007764 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.007727    4633 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea" not found
Oct 30 21:49:12.007939 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.007866    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.507843725 +0000 UTC m=+2860.857571121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea" not found
Oct 30 21:49:12.064908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:12.064874    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"]
Oct 30 21:49:12.428694 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.421636    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0ea1ea/agc-cred-00b228f0efa6: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:12.428694 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.421696    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:12.428694 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.421772    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:14.421738207 +0000 UTC m=+2862.771465597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:12.428694 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.422785    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fb2b58/agc-cred-98afef9e5c37: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:12.428694 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.422808    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:12.428694 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.422906    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials podName:620e8c62-3ca5-4bbe-a28c-dda117ece6d2 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:14.422890245 +0000 UTC m=+2862.772617632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2") : secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:12.529531 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.529495    4633 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea" not found
Oct 30 21:49:12.529814 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:12.529797    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:13.52977654 +0000 UTC m=+2861.879503928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea" not found
Oct 30 21:49:12.905899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:12.903940    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:12.932571 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:12.932543    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:12.959968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:12.959931    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:49:13.553585 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:13.553548    4633 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea" not found
Oct 30 21:49:13.554065 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:13.553634    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:15.553618559 +0000 UTC m=+2863.903345937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea" not found
Oct 30 21:49:14.497261 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:14.497223    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0ea1ea/agc-cred-00b228f0efa6: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:14.497261 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:14.497250    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:14.498134 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:14.497304    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.497288594 +0000 UTC m=+2866.847015968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:14.498134 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:14.497343    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fb2b58/agc-cred-98afef9e5c37: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:14.498134 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:14.497363    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:14.498134 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:14.497418    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials podName:620e8c62-3ca5-4bbe-a28c-dda117ece6d2 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:18.49740223 +0000 UTC m=+2866.847129606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2") : secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:15.112495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.112463    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"]
Oct 30 21:49:15.113398 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.113381    4633 cpu_manager.go:414] "containerMap: removing container" podUID="18a7653d-8782-4705-b398-e89d2d9d1fb2" containerName="sim"
Oct 30 21:49:15.113446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.113399    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a7653d-8782-4705-b398-e89d2d9d1fb2" containerName="sim"
Oct 30 21:49:15.113541 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.113531    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a7653d-8782-4705-b398-e89d2d9d1fb2" containerName="sim"
Oct 30 21:49:15.119033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.119005    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.125978 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.125949    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3d9d7b\"/\"test-model-3d9d7b-kserve-self-signed-certs\""
Oct 30 21:49:15.126130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.126024    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3d9d7b\"/\"default-dockercfg-9cmpb\""
Oct 30 21:49:15.126808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.126770    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3d9d7b\"/\"openshift-service-ca.crt\""
Oct 30 21:49:15.126964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.126830    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3d9d7b\"/\"kube-root-ca.crt\""
Oct 30 21:49:15.127456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.127429    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"]
Oct 30 21:49:15.212426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.212391    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq65s\" (UniqueName: \"kubernetes.io/projected/ba5084b7-93d3-4851-92fd-af91e4ed7852-kube-api-access-dq65s\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.212611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.212459    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-tmp-dir\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.212661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.212606    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-dshm\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.212887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.212872    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5084b7-93d3-4851-92fd-af91e4ed7852-tls-certs\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.212979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.212961    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-model-cache\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.213016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.212995    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-home\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.317692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.317358    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5084b7-93d3-4851-92fd-af91e4ed7852-tls-certs\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.317692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.317438    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-model-cache\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.317692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.317469    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-home\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.317692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.317595    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq65s\" (UniqueName: \"kubernetes.io/projected/ba5084b7-93d3-4851-92fd-af91e4ed7852-kube-api-access-dq65s\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.330696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.327046    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-model-cache\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.330696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.328144    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-home\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.330696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.328730    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-tmp-dir\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.330696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.317659    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-tmp-dir\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.330696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.328898    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-dshm\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.359884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.359765    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq65s\" (UniqueName: \"kubernetes.io/projected/ba5084b7-93d3-4851-92fd-af91e4ed7852-kube-api-access-dq65s\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.368958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.368703    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-dshm\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.398694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.389443    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5084b7-93d3-4851-92fd-af91e4ed7852-tls-certs\") pod \"test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") " pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.438995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.438953    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:15.648178 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:15.648145    4633 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea" not found
Oct 30 21:49:15.648348 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:15.648228    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:19.648212684 +0000 UTC m=+2867.997940057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea" not found
Oct 30 21:49:15.797977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.797926    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" podUID="4f050773-b095-4cae-b517-d994151796d3" containerName="payload-pre-processing" containerID="cri-o://21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86" gracePeriod=25
Oct 30 21:49:15.852625 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:49:15.852596    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba5084b7_93d3_4851_92fd_af91e4ed7852.slice/crio-417dd79c9c09cd6938bedd7fad31ce1668756080224e280606d6ba79ca5bedaa WatchSource:0}: Error finding container 417dd79c9c09cd6938bedd7fad31ce1668756080224e280606d6ba79ca5bedaa: Status 404 returned error can't find the container with id 417dd79c9c09cd6938bedd7fad31ce1668756080224e280606d6ba79ca5bedaa
Oct 30 21:49:15.852792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:15.852758    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"]
Oct 30 21:49:16.059326 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:16.059287    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk" event={"ID":"ba5084b7-93d3-4851-92fd-af91e4ed7852","Type":"ContainerStarted","Data":"62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65"}
Oct 30 21:49:16.059552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:16.059341    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk" event={"ID":"ba5084b7-93d3-4851-92fd-af91e4ed7852","Type":"ContainerStarted","Data":"417dd79c9c09cd6938bedd7fad31ce1668756080224e280606d6ba79ca5bedaa"}
Oct 30 21:49:16.086969 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:16.086919    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk" podStartSLOduration=1.086902869 podStartE2EDuration="1.086902869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:16.08271228 +0000 UTC m=+2864.432439676" watchObservedRunningTime="2026-10-30 21:49:16.086902869 +0000 UTC m=+2864.436630266"
Oct 30 21:49:16.440337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:16.440299    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:17.948068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:17.948003    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" podUID="c2a459c8-1a31-402c-9927-4bff0d3d4645" containerName="payload-processing" containerID="cri-o://c6c33b6f0867781519e7a39a358c21cb8d8b1358bdc5e837cae892dbadb8ba40" gracePeriod=25
Oct 30 21:49:17.948597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:17.948230    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" podUID="620e8c62-3ca5-4bbe-a28c-dda117ece6d2" containerName="payload-processing" containerID="cri-o://284dca055e2b6e1f812dee05ca740e7df990a82fe3ed523aa1ba4ec40211d6a3" gracePeriod=25
Oct 30 21:49:18.141604 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.141559    4633 generic.go:356] "Generic (PLEG): container finished" podID="620e8c62-3ca5-4bbe-a28c-dda117ece6d2" containerID="284dca055e2b6e1f812dee05ca740e7df990a82fe3ed523aa1ba4ec40211d6a3" exitCode=0
Oct 30 21:49:18.141858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.141745    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" event={"ID":"620e8c62-3ca5-4bbe-a28c-dda117ece6d2","Type":"ContainerDied","Data":"284dca055e2b6e1f812dee05ca740e7df990a82fe3ed523aa1ba4ec40211d6a3"}
Oct 30 21:49:18.145456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.143932    4633 generic.go:356] "Generic (PLEG): container finished" podID="c2a459c8-1a31-402c-9927-4bff0d3d4645" containerID="c6c33b6f0867781519e7a39a358c21cb8d8b1358bdc5e837cae892dbadb8ba40" exitCode=0
Oct 30 21:49:18.145456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.144888    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" event={"ID":"c2a459c8-1a31-402c-9927-4bff0d3d4645","Type":"ContainerDied","Data":"c6c33b6f0867781519e7a39a358c21cb8d8b1358bdc5e837cae892dbadb8ba40"}
Oct 30 21:49:18.557748 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:18.521060    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0ea1ea/agc-cred-00b228f0efa6: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:18.558238 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:18.558209    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw: secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:18.583701 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:18.558433    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials podName:c2a459c8-1a31-402c-9927-4bff0d3d4645 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:26.558407983 +0000 UTC m=+2874.908135375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645") : secret "agc-cred-00b228f0efa6" not found
Oct 30 21:49:18.584422 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:18.547896    4633 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fb2b58/agc-cred-98afef9e5c37: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:18.584422 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:18.583972    4633 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm: secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:18.584422 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:18.584391    4633 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials podName:620e8c62-3ca5-4bbe-a28c-dda117ece6d2 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:26.584364549 +0000 UTC m=+2874.934091923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2") : secret "agc-cred-98afef9e5c37" not found
Oct 30 21:49:18.746251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.746221    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:18.765963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.765940    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:18.797690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.797645    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-kube-api-access-r8249\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-kube-api-access-r8249\") pod \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") "
Oct 30 21:49:18.797850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.797715    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-kube-api-access-qj58c\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-kube-api-access-qj58c\") pod \"c2a459c8-1a31-402c-9927-4bff0d3d4645\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") "
Oct 30 21:49:18.797850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.797801    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-routing-overlay-volume\") pod \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") "
Oct 30 21:49:18.797984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.797931    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume\") pod \"c2a459c8-1a31-402c-9927-4bff0d3d4645\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") "
Oct 30 21:49:18.798243 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.798215    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-routing-overlay-volume" pod "620e8c62-3ca5-4bbe-a28c-dda117ece6d2" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:18.798356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.798258    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume" pod "c2a459c8-1a31-402c-9927-4bff0d3d4645" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:18.798496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.798415    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-plugins-config-volume\") pod \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") "
Oct 30 21:49:18.798496 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.798458    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials\") pod \"c2a459c8-1a31-402c-9927-4bff0d3d4645\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") "
Oct 30 21:49:18.798778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.798559    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-routing-overlay-volume\") pod \"c2a459c8-1a31-402c-9927-4bff0d3d4645\" (UID: \"c2a459c8-1a31-402c-9927-4bff0d3d4645\") "
Oct 30 21:49:18.798778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.798582    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials\") pod \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\" (UID: \"620e8c62-3ca5-4bbe-a28c-dda117ece6d2\") "
Oct 30 21:49:18.800954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.800919    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-kube-api-access-r8249" pod "620e8c62-3ca5-4bbe-a28c-dda117ece6d2" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2"). InnerVolumeSpecName "kube-api-access-r8249". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:18.803285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.803254    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-plugins-config-volume" pod "620e8c62-3ca5-4bbe-a28c-dda117ece6d2" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:18.803601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.803562    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-routing-overlay-volume" pod "c2a459c8-1a31-402c-9927-4bff0d3d4645" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:18.804008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.803988    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8249\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-kube-api-access-r8249\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.804082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.804010    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.804082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.804026    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.804082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.804037    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.804082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.804046    4633 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2a459c8-1a31-402c-9927-4bff0d3d4645-routing-overlay-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.806819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.806795    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-kube-api-access-qj58c" pod "c2a459c8-1a31-402c-9927-4bff0d3d4645" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645"). InnerVolumeSpecName "kube-api-access-qj58c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:18.818155 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.818095    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials" pod "c2a459c8-1a31-402c-9927-4bff0d3d4645" (UID: "c2a459c8-1a31-402c-9927-4bff0d3d4645"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:18.824529 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.824492    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials" pod "620e8c62-3ca5-4bbe-a28c-dda117ece6d2" (UID: "620e8c62-3ca5-4bbe-a28c-dda117ece6d2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:18.905711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.905647    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.905711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.905686    4633 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/620e8c62-3ca5-4bbe-a28c-dda117ece6d2-provider-credentials\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.905711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:18.905696    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj58c\" (UniqueName: \"kubernetes.io/projected/c2a459c8-1a31-402c-9927-4bff0d3d4645-kube-api-access-qj58c\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:19.154700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.154660    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"
Oct 30 21:49:19.155164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.154688    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm" event={"ID":"620e8c62-3ca5-4bbe-a28c-dda117ece6d2","Type":"ContainerDied","Data":"82b5287e9936d33eec2b3dbb38e0ecac5f3ec16e2f28c3cb6bfcc93fd4be3530"}
Oct 30 21:49:19.155164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.154802    4633 scope.go:122] "RemoveContainer" containerID="284dca055e2b6e1f812dee05ca740e7df990a82fe3ed523aa1ba4ec40211d6a3"
Oct 30 21:49:19.156283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.156267    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"
Oct 30 21:49:19.156366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.156299    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw" event={"ID":"c2a459c8-1a31-402c-9927-4bff0d3d4645","Type":"ContainerDied","Data":"b431b7283183920c772b36c6d6841f2d5203821d10028da8783fa652d0298150"}
Oct 30 21:49:19.169441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.169417    4633 scope.go:122] "RemoveContainer" containerID="c6c33b6f0867781519e7a39a358c21cb8d8b1358bdc5e837cae892dbadb8ba40"
Oct 30 21:49:19.190746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.190722    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"]
Oct 30 21:49:19.198090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.198063    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccbspfw"]
Oct 30 21:49:19.210921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.210893    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"]
Oct 30 21:49:19.218011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.217988    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fb2b58/payload-processing-external-model-e2e-extmod-a-fb2b58-649frwrjm"]
Oct 30 21:49:19.795322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.795292    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620e8c62-3ca5-4bbe-a28c-dda117ece6d2" path="/var/lib/kubelet/pods/620e8c62-3ca5-4bbe-a28c-dda117ece6d2/volumes"
Oct 30 21:49:19.795973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:19.795956    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a459c8-1a31-402c-9927-4bff0d3d4645" path="/var/lib/kubelet/pods/c2a459c8-1a31-402c-9927-4bff0d3d4645/volumes"
Oct 30 21:49:23.067562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:23.067520    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" podUID="4f050773-b095-4cae-b517-d994151796d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:49:25.016856 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:25.016819    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:49:25.440592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:25.440559    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:25.453632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:25.453606    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:26.237210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:26.237173    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:49:26.655699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:26.654588    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:49:29.227834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:29.227786    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk" podUID="ba5084b7-93d3-4851-92fd-af91e4ed7852" containerName="main" probeResult="failure" output="Get \"https://10.133.1.15:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:49:33.104694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:33.103549    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" podUID="4f050773-b095-4cae-b517-d994151796d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:49:36.150869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.150842    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:49:36.206359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.206322    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f050773-b095-4cae-b517-d994151796d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f050773-b095-4cae-b517-d994151796d3-plugins-config-volume\") pod \"4f050773-b095-4cae-b517-d994151796d3\" (UID: \"4f050773-b095-4cae-b517-d994151796d3\") "
Oct 30 21:49:36.206565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.206414    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f050773-b095-4cae-b517-d994151796d3-kube-api-access-27xcj\" (UniqueName: \"kubernetes.io/projected/4f050773-b095-4cae-b517-d994151796d3-kube-api-access-27xcj\") pod \"4f050773-b095-4cae-b517-d994151796d3\" (UID: \"4f050773-b095-4cae-b517-d994151796d3\") "
Oct 30 21:49:36.206798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.206772    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f050773-b095-4cae-b517-d994151796d3-plugins-config-volume" pod "4f050773-b095-4cae-b517-d994151796d3" (UID: "4f050773-b095-4cae-b517-d994151796d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:36.207365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.207337    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f050773-b095-4cae-b517-d994151796d3-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:36.208799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.208769    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f050773-b095-4cae-b517-d994151796d3-kube-api-access-27xcj" pod "4f050773-b095-4cae-b517-d994151796d3" (UID: "4f050773-b095-4cae-b517-d994151796d3"). InnerVolumeSpecName "kube-api-access-27xcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:36.309742 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.309701    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27xcj\" (UniqueName: \"kubernetes.io/projected/4f050773-b095-4cae-b517-d994151796d3-kube-api-access-27xcj\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:36.528692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.523950    4633 generic.go:356] "Generic (PLEG): container finished" podID="4f050773-b095-4cae-b517-d994151796d3" containerID="21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86" exitCode=0
Oct 30 21:49:36.528692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.524371    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" event={"ID":"4f050773-b095-4cae-b517-d994151796d3","Type":"ContainerDied","Data":"21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86"}
Oct 30 21:49:36.528692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.524468    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn" event={"ID":"4f050773-b095-4cae-b517-d994151796d3","Type":"ContainerDied","Data":"0508eee384a11a3a770da1532f3d3eff4b1971a8d4dfd94000a59a9f4a608f01"}
Oct 30 21:49:36.528692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.524508    4633 scope.go:122] "RemoveContainer" containerID="21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86"
Oct 30 21:49:36.528692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.524708    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"
Oct 30 21:49:36.567745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.567718    4633 scope.go:122] "RemoveContainer" containerID="21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86"
Oct 30 21:49:36.568090 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:36.568062    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86\": container with ID starting with 21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86 not found: ID does not exist" containerID="21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86"
Oct 30 21:49:36.568229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.568100    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86"} err="failed to get container status \"21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86\": rpc error: code = NotFound desc = could not find container \"21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86\": container with ID starting with 21b88c785e31718c2d17e9831812d96edf4baefd490995139b2dc4f53862bd86 not found: ID does not exist"
Oct 30 21:49:36.570146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.570127    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"]
Oct 30 21:49:36.575319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:36.575298    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-384164-788b8bc655-cwgqn"]
Oct 30 21:49:37.816131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:37.816033    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f050773-b095-4cae-b517-d994151796d3" path="/var/lib/kubelet/pods/4f050773-b095-4cae-b517-d994151796d3/volumes"
Oct 30 21:49:46.753687 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:46.753633    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"]
Oct 30 21:49:46.754300 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:46.753921    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb" podUID="1267f93a-e77d-4c6f-b3c0-eef84202ca93" containerName="maas-api" containerID="cri-o://c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149" gracePeriod=30
Oct 30 21:49:47.444493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.444440    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"]
Oct 30 21:49:47.451369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.451332    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-384164-2s7gs"]
Oct 30 21:49:47.458701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.458657    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:49:47.468736 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.468662    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1267f93a-e77d-4c6f-b3c0-eef84202ca93-kube-api-access-l8gwh\" (UniqueName: \"kubernetes.io/projected/1267f93a-e77d-4c6f-b3c0-eef84202ca93-kube-api-access-l8gwh\") pod \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") "
Oct 30 21:49:47.468926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.468773    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1267f93a-e77d-4c6f-b3c0-eef84202ca93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1267f93a-e77d-4c6f-b3c0-eef84202ca93-openshift-service-ca\") pod \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") "
Oct 30 21:49:47.469014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.469002    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-metrics-certs\") pod \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") "
Oct 30 21:49:47.469127 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.469108    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls\") pod \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\" (UID: \"1267f93a-e77d-4c6f-b3c0-eef84202ca93\") "
Oct 30 21:49:47.469250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.469131    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1267f93a-e77d-4c6f-b3c0-eef84202ca93-openshift-service-ca" pod "1267f93a-e77d-4c6f-b3c0-eef84202ca93" (UID: "1267f93a-e77d-4c6f-b3c0-eef84202ca93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:47.469630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.469607    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1267f93a-e77d-4c6f-b3c0-eef84202ca93-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:47.471205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.471182    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1267f93a-e77d-4c6f-b3c0-eef84202ca93-kube-api-access-l8gwh" pod "1267f93a-e77d-4c6f-b3c0-eef84202ca93" (UID: "1267f93a-e77d-4c6f-b3c0-eef84202ca93"). InnerVolumeSpecName "kube-api-access-l8gwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:47.471868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.471847    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls" pod "1267f93a-e77d-4c6f-b3c0-eef84202ca93" (UID: "1267f93a-e77d-4c6f-b3c0-eef84202ca93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:47.471959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.471866    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-metrics-certs" pod "1267f93a-e77d-4c6f-b3c0-eef84202ca93" (UID: "1267f93a-e77d-4c6f-b3c0-eef84202ca93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:47.571601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.571560    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:47.571601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.571597    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1267f93a-e77d-4c6f-b3c0-eef84202ca93-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:47.571601 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.571609    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gwh\" (UniqueName: \"kubernetes.io/projected/1267f93a-e77d-4c6f-b3c0-eef84202ca93-kube-api-access-l8gwh\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:47.697219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.697129    4633 generic.go:356] "Generic (PLEG): container finished" podID="1267f93a-e77d-4c6f-b3c0-eef84202ca93" containerID="c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149" exitCode=0
Oct 30 21:49:47.697219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.697176    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"
Oct 30 21:49:47.697461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.697208    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb" event={"ID":"1267f93a-e77d-4c6f-b3c0-eef84202ca93","Type":"ContainerDied","Data":"c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149"}
Oct 30 21:49:47.697461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.697269    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb" event={"ID":"1267f93a-e77d-4c6f-b3c0-eef84202ca93","Type":"ContainerDied","Data":"480a494b9b9c09091f0e8836c67a9a7aad1cf07ba3a40ff26e8a4a5c7002c2a7"}
Oct 30 21:49:47.697461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.697297    4633 scope.go:122] "RemoveContainer" containerID="c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149"
Oct 30 21:49:47.716093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.716071    4633 scope.go:122] "RemoveContainer" containerID="c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149"
Oct 30 21:49:47.716393 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:47.716373    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149\": container with ID starting with c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149 not found: ID does not exist" containerID="c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149"
Oct 30 21:49:47.716476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.716403    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149"} err="failed to get container status \"c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149\": rpc error: code = NotFound desc = could not find container \"c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149\": container with ID starting with c12fccd0b80e3938f531ccbeb94307e1d72823e6f75673813187349693849149 not found: ID does not exist"
Oct 30 21:49:47.739461 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.739423    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"]
Oct 30 21:49:47.747858 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.747822    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-384164-68f7c6599c-87hjb"]
Oct 30 21:49:47.793173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.793145    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1267f93a-e77d-4c6f-b3c0-eef84202ca93" path="/var/lib/kubelet/pods/1267f93a-e77d-4c6f-b3c0-eef84202ca93/volumes"
Oct 30 21:49:47.793720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:47.793659    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380d1dcd-e949-4e57-97e1-49924f762d7d" path="/var/lib/kubelet/pods/380d1dcd-e949-4e57-97e1-49924f762d7d/volumes"
Oct 30 21:49:50.501639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.501605    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"]
Oct 30 21:49:50.503018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.502991    4633 cpu_manager.go:414] "containerMap: removing container" podUID="1267f93a-e77d-4c6f-b3c0-eef84202ca93" containerName="maas-api"
Oct 30 21:49:50.503125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503020    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1267f93a-e77d-4c6f-b3c0-eef84202ca93" containerName="maas-api"
Oct 30 21:49:50.503125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503045    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c2a459c8-1a31-402c-9927-4bff0d3d4645" containerName="payload-processing"
Oct 30 21:49:50.503125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503052    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a459c8-1a31-402c-9927-4bff0d3d4645" containerName="payload-processing"
Oct 30 21:49:50.503125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503082    4633 cpu_manager.go:414] "containerMap: removing container" podUID="4f050773-b095-4cae-b517-d994151796d3" containerName="payload-pre-processing"
Oct 30 21:49:50.503125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503091    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f050773-b095-4cae-b517-d994151796d3" containerName="payload-pre-processing"
Oct 30 21:49:50.503296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503157    4633 cpu_manager.go:414] "containerMap: removing container" podUID="620e8c62-3ca5-4bbe-a28c-dda117ece6d2" containerName="payload-processing"
Oct 30 21:49:50.503296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503165    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620e8c62-3ca5-4bbe-a28c-dda117ece6d2" containerName="payload-processing"
Oct 30 21:49:50.503378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503360    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="620e8c62-3ca5-4bbe-a28c-dda117ece6d2" containerName="payload-processing"
Oct 30 21:49:50.503414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503391    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="1267f93a-e77d-4c6f-b3c0-eef84202ca93" containerName="maas-api"
Oct 30 21:49:50.503414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503403    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a459c8-1a31-402c-9927-4bff0d3d4645" containerName="payload-processing"
Oct 30 21:49:50.503478 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.503418    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f050773-b095-4cae-b517-d994151796d3" containerName="payload-pre-processing"
Oct 30 21:49:50.519697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.516601    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"]
Oct 30 21:49:50.519697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.516784    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.524039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.522330    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-adf308-serving-cert\""
Oct 30 21:49:50.542751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.542721    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676be30a-0353-4bbf-8775-1219db088582-openshift-service-ca\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.542950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.542768    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-maas-api-tls\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.543176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.543120    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wg26\" (UniqueName: \"kubernetes.io/projected/676be30a-0353-4bbf-8775-1219db088582-kube-api-access-8wg26\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.543176 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.543169    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-metrics-certs\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.645328 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.645294    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wg26\" (UniqueName: \"kubernetes.io/projected/676be30a-0353-4bbf-8775-1219db088582-kube-api-access-8wg26\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.645517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.645345    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-metrics-certs\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.645517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.645449    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676be30a-0353-4bbf-8775-1219db088582-openshift-service-ca\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.645517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.645472    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-maas-api-tls\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.646171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.646147    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676be30a-0353-4bbf-8775-1219db088582-openshift-service-ca\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.648122 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.648090    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-metrics-certs\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.648234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.648136    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-maas-api-tls\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.654361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.654333    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wg26\" (UniqueName: \"kubernetes.io/projected/676be30a-0353-4bbf-8775-1219db088582-kube-api-access-8wg26\") pod \"maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:50.845148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:50.845072    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:51.356045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:51.355615    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"]
Oct 30 21:49:51.366042 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:49:51.366008    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676be30a_0353_4bbf_8775_1219db088582.slice/crio-57d7e2be68c10f1ed2ba646cf0d33f0f42c51969e66af40c1c4a08b571d3c096 WatchSource:0}: Error finding container 57d7e2be68c10f1ed2ba646cf0d33f0f42c51969e66af40c1c4a08b571d3c096: Status 404 returned error can't find the container with id 57d7e2be68c10f1ed2ba646cf0d33f0f42c51969e66af40c1c4a08b571d3c096
Oct 30 21:49:51.812956 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:51.812926    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k" event={"ID":"676be30a-0353-4bbf-8775-1219db088582","Type":"ContainerStarted","Data":"57d7e2be68c10f1ed2ba646cf0d33f0f42c51969e66af40c1c4a08b571d3c096"}
Oct 30 21:49:52.082104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.082031    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"]
Oct 30 21:49:52.087928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.087907    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.105983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.105950    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"]
Oct 30 21:49:52.207495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.207460    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.207778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.207532    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.207778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.207768    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd525\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-kube-api-access-jd525\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.207946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.207896    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.207946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.207939    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.208061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.207971    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.208061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.207996    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.208061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.208026    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45d142-8150-4eec-9dbb-58e94d3d171d-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.208233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.208109    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310100 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310064    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310121    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310164    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310190    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310212    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45d142-8150-4eec-9dbb-58e94d3d171d-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310291 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310248    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310517    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310611    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310619    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.310849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.310729    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd525\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-kube-api-access-jd525\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.311296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.311218    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.311296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.311244    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.311816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.311401    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.311816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.311585    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45d142-8150-4eec-9dbb-58e94d3d171d-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.313306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.313284    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.313575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.313556    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.320221 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.320196    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.321792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.321770    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd525\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-kube-api-access-jd525\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.405184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.405151    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:52.716626 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:49:52.716583    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f45d142_8150_4eec_9dbb_58e94d3d171d.slice/crio-382adbd31b2fd5cf2eaa473ea31db06d8ce62a000bb463112a714888f04e532f WatchSource:0}: Error finding container 382adbd31b2fd5cf2eaa473ea31db06d8ce62a000bb463112a714888f04e532f: Status 404 returned error can't find the container with id 382adbd31b2fd5cf2eaa473ea31db06d8ce62a000bb463112a714888f04e532f
Oct 30 21:49:52.717094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.717074    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"]
Oct 30 21:49:52.719550 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.719515    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:49:52.719658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.719585    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:49:52.719658 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.719613    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:49:52.869431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.869393    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k" event={"ID":"676be30a-0353-4bbf-8775-1219db088582","Type":"ContainerStarted","Data":"94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c"}
Oct 30 21:49:52.870004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.869489    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:52.872003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.871976    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" event={"ID":"6f45d142-8150-4eec-9dbb-58e94d3d171d","Type":"ContainerStarted","Data":"382adbd31b2fd5cf2eaa473ea31db06d8ce62a000bb463112a714888f04e532f"}
Oct 30 21:49:52.900144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:52.900062    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k" podStartSLOduration=2.419346108 podStartE2EDuration="2.90004457s" totalImagesPullingTime="480.698462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:52.898548589 +0000 UTC m=+2901.248275986" watchObservedRunningTime="2026-10-30 21:49:52.90004457 +0000 UTC m=+2901.249771968"
Oct 30 21:49:53.882979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:53.882934    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" event={"ID":"6f45d142-8150-4eec-9dbb-58e94d3d171d","Type":"ContainerStarted","Data":"d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495"}
Oct 30 21:49:53.940608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:53.937401    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podStartSLOduration=1.937379726 podStartE2EDuration="1.937379726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:53.922183312 +0000 UTC m=+2902.271910708" watchObservedRunningTime="2026-10-30 21:49:53.937379726 +0000 UTC m=+2902.287107123"
Oct 30 21:49:54.405544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:54.405513    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:49:54.407512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:54.407482    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:49:54.407644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:54.407542    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 21:49:55.052113 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:49:55.052073    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:49:55.419700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:55.416253    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:49:55.419700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:55.416317    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 21:49:55.475245 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:55.475153    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"]
Oct 30 21:49:55.475444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:55.475404    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy" containerID="cri-o://e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32" gracePeriod=30
Oct 30 21:49:56.405628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:56.405597    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:49:56.406043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:56.405651    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 21:49:57.406240 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:57.406204    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:49:57.406731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:57.406262    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 21:49:58.406475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:58.406440    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:49:58.406932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:58.406493    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 21:49:58.889211 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:58.889132    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:49:59.405652 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:59.405617    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:49:59.405867 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:49:59.405687    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Oct 30 21:50:00.411118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.411089    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:50:00.411589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.411517    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:50:00.412178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.412162    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:50:00.798235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.798206    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:50:00.814522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814494    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-certs\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.814724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814538    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-credential-socket\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.814724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814643    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-socket\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.814870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814740    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45f95ab-4f37-4976-8512-0fc08613ecef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45f95ab-4f37-4976-8512-0fc08613ecef-istiod-ca-cert\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.814870 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814808    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-certs" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.814976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814881    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-kube-api-access-dp98m\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-kube-api-access-dp98m\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.814976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814924    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-credential-socket" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.814976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814957    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-data\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.814976 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.814969    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-socket" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.815181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.815009    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-podinfo\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.815181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.815068    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-envoy\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.815287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.815181    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-token\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-token\") pod \"c45f95ab-4f37-4976-8512-0fc08613ecef\" (UID: \"c45f95ab-4f37-4976-8512-0fc08613ecef\") "
Oct 30 21:50:00.815287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.815228    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45f95ab-4f37-4976-8512-0fc08613ecef-istiod-ca-cert" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:00.815392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.815313    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-data" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.816414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.816388    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.821705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.821620    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.821954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.821938    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.822080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.822066    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.822183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.817927    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-envoy" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.822241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.819616    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-token" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:00.822241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.820388    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-podinfo" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:50:00.822241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.822149    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-kube-api-access-dp98m" pod "c45f95ab-4f37-4976-8512-0fc08613ecef" (UID: "c45f95ab-4f37-4976-8512-0fc08613ecef"). InnerVolumeSpecName "kube-api-access-dp98m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:00.822417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.822403    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c45f95ab-4f37-4976-8512-0fc08613ecef-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.924832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.924754    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp98m\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-kube-api-access-dp98m\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.924832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.924784    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.924832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.924794    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.924832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:00.924805    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c45f95ab-4f37-4976-8512-0fc08613ecef-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:01.156939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.156899    4633 generic.go:356] "Generic (PLEG): container finished" podID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerID="e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32" exitCode=0
Oct 30 21:50:01.157152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.156948    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"
Oct 30 21:50:01.157152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.156981    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" event={"ID":"c45f95ab-4f37-4976-8512-0fc08613ecef","Type":"ContainerDied","Data":"e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32"}
Oct 30 21:50:01.157152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.157062    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc" event={"ID":"c45f95ab-4f37-4976-8512-0fc08613ecef","Type":"ContainerDied","Data":"1dbf51a0eaac5f0693823680fe2c38cc01765c2c9b9f4636ce7604fada67ec51"}
Oct 30 21:50:01.157152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.157093    4633 scope.go:122] "RemoveContainer" containerID="e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32"
Oct 30 21:50:01.171113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.171090    4633 scope.go:122] "RemoveContainer" containerID="e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32"
Oct 30 21:50:01.171449 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:50:01.171428    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32\": container with ID starting with e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32 not found: ID does not exist" containerID="e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32"
Oct 30 21:50:01.171509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.171459    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32"} err="failed to get container status \"e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32\": rpc error: code = NotFound desc = could not find container \"e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32\": container with ID starting with e3ec605640f076ef68c026cb6d2b1e5aca667b68775ae0f69c04a2c3f8613f32 not found: ID does not exist"
Oct 30 21:50:01.184785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.184642    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"]
Oct 30 21:50:01.190044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.190019    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-lglfc"]
Oct 30 21:50:01.796415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:01.796375    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" path="/var/lib/kubelet/pods/c45f95ab-4f37-4976-8512-0fc08613ecef/volumes"
Oct 30 21:50:18.611542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.611457    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"]
Oct 30 21:50:18.612745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.612715    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy"
Oct 30 21:50:18.612745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.612744    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy"
Oct 30 21:50:18.613011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.612995    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45f95ab-4f37-4976-8512-0fc08613ecef" containerName="istio-proxy"
Oct 30 21:50:18.618798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.618779    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.632152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.632118    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"]
Oct 30 21:50:18.636034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.636002    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e8d315-openshift-default-dockercfg-7c42t\""
Oct 30 21:50:18.741333 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.741289    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-credential-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.741763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.741738    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-certs\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.742038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.742022    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-token\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.743374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.743346    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-podinfo\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.797695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.743650    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-envoy\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.797695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.796869    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-data\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.797695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.796987    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.797695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.797072    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510937e8-5b5e-474d-9d5b-452eb5062cb8-istiod-ca-cert\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.797695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.797172    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbb6\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-kube-api-access-hvbb6\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.902829    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-token\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.902886    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-podinfo\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.903001    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-envoy\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.903075    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-data\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.903172    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.903244    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510937e8-5b5e-474d-9d5b-452eb5062cb8-istiod-ca-cert\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.903344    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbb6\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-kube-api-access-hvbb6\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.903384    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-credential-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.903474    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-certs\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.908753 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.904034    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-certs\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.909483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.909201    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.928030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.915106    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-podinfo\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.928295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.926456    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-data\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.928453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.927334    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510937e8-5b5e-474d-9d5b-452eb5062cb8-istiod-ca-cert\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.928586 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.927786    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-credential-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.942514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.942475    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-token\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.975351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.975090    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-envoy\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:18.976233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:18.976208    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbb6\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-kube-api-access-hvbb6\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:19.245231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:19.245145    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:19.542293 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:50:19.542263    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510937e8_5b5e_474d_9d5b_452eb5062cb8.slice/crio-5448b56912733f6a680f8b7e799fc72ab0df467eac4ba1a72d59e47f3a10ea20 WatchSource:0}: Error finding container 5448b56912733f6a680f8b7e799fc72ab0df467eac4ba1a72d59e47f3a10ea20: Status 404 returned error can't find the container with id 5448b56912733f6a680f8b7e799fc72ab0df467eac4ba1a72d59e47f3a10ea20
Oct 30 21:50:19.543078 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:19.543047    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"]
Oct 30 21:50:19.545444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:19.545407    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:50:19.545562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:19.545496    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:50:19.545562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:19.545544    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:50:20.462843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:20.462800    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" event={"ID":"510937e8-5b5e-474d-9d5b-452eb5062cb8","Type":"ContainerStarted","Data":"75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec"}
Oct 30 21:50:20.463285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:20.462874    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" event={"ID":"510937e8-5b5e-474d-9d5b-452eb5062cb8","Type":"ContainerStarted","Data":"5448b56912733f6a680f8b7e799fc72ab0df467eac4ba1a72d59e47f3a10ea20"}
Oct 30 21:50:20.489706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:20.489637    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" podStartSLOduration=2.489621201 podStartE2EDuration="2.489621201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:20.48311169 +0000 UTC m=+2928.832839087" watchObservedRunningTime="2026-10-30 21:50:20.489621201 +0000 UTC m=+2928.839348596"
Oct 30 21:50:21.245752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:21.245714    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:21.252483 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:21.252453    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:21.471390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:21.471358    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:21.472371 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:21.472354    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:50:25.016632 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:50:25.016591    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:50:46.815608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:50:46.815574    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:50:55.017218 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:50:55.017173    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:51:22.220777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:22.220724    4633 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": read tcp 10.133.0.2:55684->10.133.1.18:15021: read: connection reset by peer" start-of-body=
Oct 30 21:51:22.221224 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:22.220796    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": read tcp 10.133.0.2:55684->10.133.1.18:15021: read: connection reset by peer"
Oct 30 21:51:22.509354 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:22.509269    4633 generic.go:356] "Generic (PLEG): container finished" podID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerID="75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec" exitCode=0
Oct 30 21:51:22.509511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:22.509344    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" event={"ID":"510937e8-5b5e-474d-9d5b-452eb5062cb8","Type":"ContainerDied","Data":"75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec"}
Oct 30 21:51:22.509938 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:22.509923    4633 scope.go:122] "RemoveContainer" containerID="75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec"
Oct 30 21:51:22.512032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:22.512003    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:51:22.512133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:22.512085    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:51:22.512133 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:22.512117    4633 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 30 21:51:23.545746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:23.545711    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" event={"ID":"510937e8-5b5e-474d-9d5b-452eb5062cb8","Type":"ContainerStarted","Data":"a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27"}
Oct 30 21:51:24.246086 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:24.246049    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:51:24.246285 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:24.246266    4633 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 21:51:24.246346 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:24.246314    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 21:51:25.044418 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:51:25.044379    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:51:25.246104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:25.246060    4633 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 21:51:25.246286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:25.246120    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 21:51:26.246298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:26.246262    4633 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 21:51:26.246778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:26.246323    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 21:51:27.246264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.246222    4633 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 21:51:27.246491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.246284    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 21:51:27.844799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.844690    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"]
Oct 30 21:51:27.851923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.851881    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"]
Oct 30 21:51:27.852256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.852196    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:27.855906 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.855888    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e8d315\"/\"test-model-e8d315-kserve-self-signed-certs\""
Oct 30 21:51:27.856915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.856897    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e8d315\"/\"kube-root-ca.crt\""
Oct 30 21:51:27.857188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.857172    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e8d315\"/\"default-dockercfg-wq98n\""
Oct 30 21:51:27.857443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.857396    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e8d315\"/\"openshift-service-ca.crt\""
Oct 30 21:51:27.951812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.951543    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9645737c-61cc-475d-9832-d5dda1b7dee9-tls-certs\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:27.951812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.951587    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-dshm\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:27.952222 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.952199    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-tmp-dir\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:27.952327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.952309    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-model-cache\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:27.952485 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.952471    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9758w\" (UniqueName: \"kubernetes.io/projected/9645737c-61cc-475d-9832-d5dda1b7dee9-kube-api-access-9758w\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:27.952715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:27.952689    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-home\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.055579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.055542    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9645737c-61cc-475d-9832-d5dda1b7dee9-tls-certs\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.055963 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.055939    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-dshm\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.056223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.056207    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-tmp-dir\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.069694 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.056418    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-model-cache\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.070004 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.069976    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9758w\" (UniqueName: \"kubernetes.io/projected/9645737c-61cc-475d-9832-d5dda1b7dee9-kube-api-access-9758w\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.070323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.070305    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-home\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.072883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.072105    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-home\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.091027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.090992    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-tmp-dir\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.091895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.091870    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-model-cache\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.092261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.092232    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-dshm\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.114749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.111379    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9758w\" (UniqueName: \"kubernetes.io/projected/9645737c-61cc-475d-9832-d5dda1b7dee9-kube-api-access-9758w\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.171712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.167188    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9645737c-61cc-475d-9832-d5dda1b7dee9-tls-certs\") pod \"test-model-e8d315-kserve-556d99995-qh6v2\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") " pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.175699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.173419    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:28.253283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.253248    4633 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 21:51:28.254804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.254769    4633 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 21:51:28.575998 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:51:28.575960    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9645737c_61cc_475d_9832_d5dda1b7dee9.slice/crio-2701fdda90e910a32b34917e64e42e38cd48bb3477bee2b33c0067749aa1517c WatchSource:0}: Error finding container 2701fdda90e910a32b34917e64e42e38cd48bb3477bee2b33c0067749aa1517c: Status 404 returned error can't find the container with id 2701fdda90e910a32b34917e64e42e38cd48bb3477bee2b33c0067749aa1517c
Oct 30 21:51:28.579531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.579505    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"]
Oct 30 21:51:28.756319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:28.756285    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2" event={"ID":"9645737c-61cc-475d-9832-d5dda1b7dee9","Type":"ContainerStarted","Data":"2701fdda90e910a32b34917e64e42e38cd48bb3477bee2b33c0067749aa1517c"}
Oct 30 21:51:29.250337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:29.250307    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:51:29.250806 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:29.250784    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:51:29.251557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:29.251539    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:51:29.768828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:29.768788    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2" event={"ID":"9645737c-61cc-475d-9832-d5dda1b7dee9","Type":"ContainerStarted","Data":"fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948"}
Oct 30 21:51:29.813901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:29.813846    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2" podStartSLOduration=2.813829678 podStartE2EDuration="2.813829678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:29.784766345 +0000 UTC m=+2998.134493741" watchObservedRunningTime="2026-10-30 21:51:29.813829678 +0000 UTC m=+2998.163557077"
Oct 30 21:51:30.185697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:30.182779    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:33.376457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:33.376417    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:51:33.391817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:33.391785    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:51:33.395875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:33.395853    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:51:33.410708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:33.410664    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:51:38.174481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:38.174443    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:38.187357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:38.187331    4633 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:38.984575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:38.984539    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:51:39.183439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:51:39.183405    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:51:55.017077 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:51:55.016992    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:52:03.971883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:03.971846    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"]
Oct 30 21:52:03.972330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:03.972105    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy" containerID="cri-o://b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898" gracePeriod=30
Oct 30 21:52:05.624312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:05.624275    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"]
Oct 30 21:52:05.624784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:05.624480    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy" containerID="cri-o://3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631" gracePeriod=30
Oct 30 21:52:09.344250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.344221    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:52:09.455347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455310    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-data\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.455544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455472    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-envoy\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.455609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455560    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-credential-socket\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.455660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455610    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-istio-token\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-istio-token\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.455660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455617    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-data" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:09.455763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455659    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae638011-2505-459d-ad37-e3e618d710dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae638011-2505-459d-ad37-e3e618d710dc-istio-podinfo\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.455799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455780    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-certs\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.455875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455843    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-credential-socket" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:09.455875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455867    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-socket\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.456109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.455988    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-kube-api-access-79px5\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-kube-api-access-79px5\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.456109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.456074    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae638011-2505-459d-ad37-e3e618d710dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae638011-2505-459d-ad37-e3e618d710dc-istiod-ca-cert\") pod \"ae638011-2505-459d-ad37-e3e618d710dc\" (UID: \"ae638011-2505-459d-ad37-e3e618d710dc\") "
Oct 30 21:52:09.456109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.456076    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-certs" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:09.456310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.456172    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-socket" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:09.456535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.456516    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae638011-2505-459d-ad37-e3e618d710dc-istiod-ca-cert" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:09.457612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.457586    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.457748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.457620    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.457748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.457637    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.457748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.457653    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae638011-2505-459d-ad37-e3e618d710dc-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.457748 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.457718    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.458425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.458400    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-kube-api-access-79px5" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "kube-api-access-79px5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:09.458558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.458487    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-istio-token" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:09.458558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.458546    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae638011-2505-459d-ad37-e3e618d710dc-istio-podinfo" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:52:09.458641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.458625    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-envoy" pod "ae638011-2505-459d-ad37-e3e618d710dc" (UID: "ae638011-2505-459d-ad37-e3e618d710dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:09.512937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.512896    4633 generic.go:356] "Generic (PLEG): container finished" podID="ae638011-2505-459d-ad37-e3e618d710dc" containerID="b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898" exitCode=0
Oct 30 21:52:09.513140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.512945    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"
Oct 30 21:52:09.513140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.512979    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" event={"ID":"ae638011-2505-459d-ad37-e3e618d710dc","Type":"ContainerDied","Data":"b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898"}
Oct 30 21:52:09.513140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.513050    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb" event={"ID":"ae638011-2505-459d-ad37-e3e618d710dc","Type":"ContainerDied","Data":"edda11b70510327a3b182d85e3d35b395cd16eae9274c5222b0239bf2d967f21"}
Oct 30 21:52:09.513140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.513080    4633 scope.go:122] "RemoveContainer" containerID="b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898"
Oct 30 21:52:09.529593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.529570    4633 scope.go:122] "RemoveContainer" containerID="18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43"
Oct 30 21:52:09.546124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.546089    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"]
Oct 30 21:52:09.546744 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.546602    4633 scope.go:122] "RemoveContainer" containerID="b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898"
Oct 30 21:52:09.547020 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:52:09.546992    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898\": container with ID starting with b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898 not found: ID does not exist" containerID="b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898"
Oct 30 21:52:09.547112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.547034    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898"} err="failed to get container status \"b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898\": rpc error: code = NotFound desc = could not find container \"b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898\": container with ID starting with b1d01f78b395e15a285a34bd514ad17d6109026fb9df219b2835df639cf8b898 not found: ID does not exist"
Oct 30 21:52:09.547112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.547059    4633 scope.go:122] "RemoveContainer" containerID="18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43"
Oct 30 21:52:09.547412 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:52:09.547388    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43\": container with ID starting with 18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43 not found: ID does not exist" containerID="18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43"
Oct 30 21:52:09.547502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.547420    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43"} err="failed to get container status \"18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43\": rpc error: code = NotFound desc = could not find container \"18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43\": container with ID starting with 18d315f687045974e03cf709241f04ccfe887f9bdcd29396d5abb0178a476b43 not found: ID does not exist"
Oct 30 21:52:09.551778 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.551758    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-qr2zb"]
Oct 30 21:52:09.559782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.559723    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79px5\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-kube-api-access-79px5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.559782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.559749    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae638011-2505-459d-ad37-e3e618d710dc-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.559782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.559760    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae638011-2505-459d-ad37-e3e618d710dc-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.559782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.559768    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae638011-2505-459d-ad37-e3e618d710dc-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.794316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:09.794287    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae638011-2505-459d-ad37-e3e618d710dc" path="/var/lib/kubelet/pods/ae638011-2505-459d-ad37-e3e618d710dc/volumes"
Oct 30 21:52:10.497275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:10.497240    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:52:10.497780 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:10.497299    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:52:11.012623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.012564    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:52:11.206855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.206820    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-envoy\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.207036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.206912    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-certs\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.207036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.206944    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-kube-api-access-dt2kr\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-kube-api-access-dt2kr\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.207036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.207019    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-data\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.207213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.207052    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-istio-token\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-istio-token\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.207273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.207228    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-certs" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:11.207335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.207306    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-data" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:11.207621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.207563    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-credential-socket\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.208096 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.208069    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-credential-socket" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:11.208223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.208202    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7c630ce-a087-471a-8916-1d97307638fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c630ce-a087-471a-8916-1d97307638fd-istio-podinfo\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.210739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.209867    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-envoy" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:11.210739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.209928    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-kube-api-access-dt2kr" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "kube-api-access-dt2kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:11.210739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.210295    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c630ce-a087-471a-8916-1d97307638fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c630ce-a087-471a-8916-1d97307638fd-istiod-ca-cert\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.210739 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.210351    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-socket\") pod \"b7c630ce-a087-471a-8916-1d97307638fd\" (UID: \"b7c630ce-a087-471a-8916-1d97307638fd\") "
Oct 30 21:52:11.212214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.211171    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7c630ce-a087-471a-8916-1d97307638fd-istio-podinfo" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:52:11.212214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.211343    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c630ce-a087-471a-8916-1d97307638fd-istiod-ca-cert" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:11.212214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.211382    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-istio-token" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:11.212661 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.211489    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-socket" pod "b7c630ce-a087-471a-8916-1d97307638fd" (UID: "b7c630ce-a087-471a-8916-1d97307638fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215192    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c630ce-a087-471a-8916-1d97307638fd-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215220    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c630ce-a087-471a-8916-1d97307638fd-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215235    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215251    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215260    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215271    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt2kr\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-kube-api-access-dt2kr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215280    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215294    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c630ce-a087-471a-8916-1d97307638fd-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.215975 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.215309    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c630ce-a087-471a-8916-1d97307638fd-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:11.535898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.535868    4633 generic.go:356] "Generic (PLEG): container finished" podID="b7c630ce-a087-471a-8916-1d97307638fd" containerID="3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631" exitCode=0
Oct 30 21:52:11.536381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.535934    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"
Oct 30 21:52:11.536381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.535956    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" event={"ID":"b7c630ce-a087-471a-8916-1d97307638fd","Type":"ContainerDied","Data":"3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631"}
Oct 30 21:52:11.536381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.536027    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt" event={"ID":"b7c630ce-a087-471a-8916-1d97307638fd","Type":"ContainerDied","Data":"379ab3c89508313c6c48d15f196d80af6a756dc9428694611d010fa950f4143e"}
Oct 30 21:52:11.536381 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.536061    4633 scope.go:122] "RemoveContainer" containerID="3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631"
Oct 30 21:52:11.552283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.552262    4633 scope.go:122] "RemoveContainer" containerID="3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631"
Oct 30 21:52:11.552641 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:52:11.552611    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631\": container with ID starting with 3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631 not found: ID does not exist" containerID="3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631"
Oct 30 21:52:11.552768 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.552650    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631"} err="failed to get container status \"3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631\": rpc error: code = NotFound desc = could not find container \"3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631\": container with ID starting with 3e88f5eb6950324c338cf56919ac5710c7373cf26c13d0ec1cfa6b8a0490f631 not found: ID does not exist"
Oct 30 21:52:11.567104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.567079    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"]
Oct 30 21:52:11.574111 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.574087    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-jg4qt"]
Oct 30 21:52:11.802159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:11.802067    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c630ce-a087-471a-8916-1d97307638fd" path="/var/lib/kubelet/pods/b7c630ce-a087-471a-8916-1d97307638fd/volumes"
Oct 30 21:52:16.090446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:16.090404    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"]
Oct 30 21:52:21.698946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:21.698909    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:52:25.066493 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:52:25.066453    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:52:31.128902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.128843    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj" podUID="7b0d20f7-3a1a-4875-84fb-29ae05de3304" containerName="main" containerID="cri-o://cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73" gracePeriod=45
Oct 30 21:52:31.463063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.463024    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:52:31.527705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.522787    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0d20f7-3a1a-4875-84fb-29ae05de3304-kube-api-access-6sxwf\" (UniqueName: \"kubernetes.io/projected/7b0d20f7-3a1a-4875-84fb-29ae05de3304-kube-api-access-6sxwf\") pod \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") "
Oct 30 21:52:31.527705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.522851    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-home\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-home\") pod \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") "
Oct 30 21:52:31.527705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.522973    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-model-cache\") pod \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") "
Oct 30 21:52:31.527705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.523098    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tls-certs\") pod \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") "
Oct 30 21:52:31.527705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.523155    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tmp-dir\") pod \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") "
Oct 30 21:52:31.527705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.523461    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-dshm\") pod \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\" (UID: \"7b0d20f7-3a1a-4875-84fb-29ae05de3304\") "
Oct 30 21:52:31.549918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.539206    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-model-cache" pod "7b0d20f7-3a1a-4875-84fb-29ae05de3304" (UID: "7b0d20f7-3a1a-4875-84fb-29ae05de3304"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:31.549918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.539598    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-home" pod "7b0d20f7-3a1a-4875-84fb-29ae05de3304" (UID: "7b0d20f7-3a1a-4875-84fb-29ae05de3304"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:31.549918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.539834    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tmp-dir" pod "7b0d20f7-3a1a-4875-84fb-29ae05de3304" (UID: "7b0d20f7-3a1a-4875-84fb-29ae05de3304"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:31.550236 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.549955    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0d20f7-3a1a-4875-84fb-29ae05de3304-kube-api-access-6sxwf" pod "7b0d20f7-3a1a-4875-84fb-29ae05de3304" (UID: "7b0d20f7-3a1a-4875-84fb-29ae05de3304"). InnerVolumeSpecName "kube-api-access-6sxwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:31.551899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.551845    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tls-certs" pod "7b0d20f7-3a1a-4875-84fb-29ae05de3304" (UID: "7b0d20f7-3a1a-4875-84fb-29ae05de3304"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:31.558590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.558537    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-dshm" pod "7b0d20f7-3a1a-4875-84fb-29ae05de3304" (UID: "7b0d20f7-3a1a-4875-84fb-29ae05de3304"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:31.625643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.625597    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:31.625643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.625631    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sxwf\" (UniqueName: \"kubernetes.io/projected/7b0d20f7-3a1a-4875-84fb-29ae05de3304-kube-api-access-6sxwf\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:31.625643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.625644    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:31.625643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.625653    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:31.626055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.625662    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:31.626055 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.625722    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0d20f7-3a1a-4875-84fb-29ae05de3304-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:31.744046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.743962    4633 generic.go:356] "Generic (PLEG): container finished" podID="7b0d20f7-3a1a-4875-84fb-29ae05de3304" containerID="cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73" exitCode=0
Oct 30 21:52:31.744046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.744015    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"
Oct 30 21:52:31.744046 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.744026    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj" event={"ID":"7b0d20f7-3a1a-4875-84fb-29ae05de3304","Type":"ContainerDied","Data":"cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73"}
Oct 30 21:52:31.744340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.744062    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj" event={"ID":"7b0d20f7-3a1a-4875-84fb-29ae05de3304","Type":"ContainerDied","Data":"72aae223e15ba7af1e583d48395dc632157b6e22272f6a282f4bfbc3c7c741d0"}
Oct 30 21:52:31.744340 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.744080    4633 scope.go:122] "RemoveContainer" containerID="cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73"
Oct 30 21:52:31.756125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.756108    4633 scope.go:122] "RemoveContainer" containerID="cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73"
Oct 30 21:52:31.756419 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:52:31.756396    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73\": container with ID starting with cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73 not found: ID does not exist" containerID="cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73"
Oct 30 21:52:31.756462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.756430    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73"} err="failed to get container status \"cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73\": rpc error: code = NotFound desc = could not find container \"cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73\": container with ID starting with cbd33db3a0b6f61ef5631133e80f16e0cf83dec3e557589ebf0a7c1f3d911f73 not found: ID does not exist"
Oct 30 21:52:31.773269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.773235    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"]
Oct 30 21:52:31.778140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.778114    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8431de/test-model-8431de-kserve-5549fc66b9-njzzj"]
Oct 30 21:52:31.798104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:31.798059    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0d20f7-3a1a-4875-84fb-29ae05de3304" path="/var/lib/kubelet/pods/7b0d20f7-3a1a-4875-84fb-29ae05de3304/volumes"
Oct 30 21:52:37.737044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:37.737011    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"]
Oct 30 21:52:37.779994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:37.779957    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"]
Oct 30 21:52:42.737380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:42.737327    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz" podUID="49fddf86-5424-4a8a-b896-ca1aafc24165" containerName="payload-processing" containerID="cri-o://154bc56a006f37a6417d319b9855d96847f89d8cba7e9b486c2f30c5c0c52e2c" gracePeriod=25
Oct 30 21:52:42.780826 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:42.780791    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt" podUID="e42295e3-7338-4654-ae73-c08cfa9a4875" containerName="payload-pre-processing" containerID="cri-o://7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1" gracePeriod=25
Oct 30 21:52:42.872329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:42.872294    4633 generic.go:356] "Generic (PLEG): container finished" podID="49fddf86-5424-4a8a-b896-ca1aafc24165" containerID="154bc56a006f37a6417d319b9855d96847f89d8cba7e9b486c2f30c5c0c52e2c" exitCode=0
Oct 30 21:52:42.872329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:42.872316    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz" event={"ID":"49fddf86-5424-4a8a-b896-ca1aafc24165","Type":"ContainerDied","Data":"154bc56a006f37a6417d319b9855d96847f89d8cba7e9b486c2f30c5c0c52e2c"}
Oct 30 21:52:43.173051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.173024    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:52:43.306584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.306545    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49fddf86-5424-4a8a-b896-ca1aafc24165-kube-api-access-p5tr8\" (UniqueName: \"kubernetes.io/projected/49fddf86-5424-4a8a-b896-ca1aafc24165-kube-api-access-p5tr8\") pod \"49fddf86-5424-4a8a-b896-ca1aafc24165\" (UID: \"49fddf86-5424-4a8a-b896-ca1aafc24165\") "
Oct 30 21:52:43.307003 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.306985    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49fddf86-5424-4a8a-b896-ca1aafc24165-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fddf86-5424-4a8a-b896-ca1aafc24165-plugins-config-volume\") pod \"49fddf86-5424-4a8a-b896-ca1aafc24165\" (UID: \"49fddf86-5424-4a8a-b896-ca1aafc24165\") "
Oct 30 21:52:43.307334 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.307311    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fddf86-5424-4a8a-b896-ca1aafc24165-plugins-config-volume" pod "49fddf86-5424-4a8a-b896-ca1aafc24165" (UID: "49fddf86-5424-4a8a-b896-ca1aafc24165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:43.307644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.307628    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49fddf86-5424-4a8a-b896-ca1aafc24165-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:43.309121 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.309100    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fddf86-5424-4a8a-b896-ca1aafc24165-kube-api-access-p5tr8" pod "49fddf86-5424-4a8a-b896-ca1aafc24165" (UID: "49fddf86-5424-4a8a-b896-ca1aafc24165"). InnerVolumeSpecName "kube-api-access-p5tr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:43.409061 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.409027    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5tr8\" (UniqueName: \"kubernetes.io/projected/49fddf86-5424-4a8a-b896-ca1aafc24165-kube-api-access-p5tr8\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:43.882621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.882589    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"
Oct 30 21:52:43.882621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.882604    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz" event={"ID":"49fddf86-5424-4a8a-b896-ca1aafc24165","Type":"ContainerDied","Data":"e4a1d6d787a1a7d84bc9a0a80b86adba0fa6f0beb732d910d8aa83feb31fa9d8"}
Oct 30 21:52:43.883167 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.882712    4633 scope.go:122] "RemoveContainer" containerID="154bc56a006f37a6417d319b9855d96847f89d8cba7e9b486c2f30c5c0c52e2c"
Oct 30 21:52:43.904404 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.904377    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"]
Oct 30 21:52:43.911278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.911253    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8431de-5cd9475675-pv7xz"]
Oct 30 21:52:43.920233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:43.920208    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt" podUID="e42295e3-7338-4654-ae73-c08cfa9a4875" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:52:45.795175 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:45.795141    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49fddf86-5424-4a8a-b896-ca1aafc24165" path="/var/lib/kubelet/pods/49fddf86-5424-4a8a-b896-ca1aafc24165/volumes"
Oct 30 21:52:53.919515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:53.919464    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt" podUID="e42295e3-7338-4654-ae73-c08cfa9a4875" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:52:55.019440 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:52:55.019407    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:52:58.580152 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:52:58.580116    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"]
Oct 30 21:53:03.097251 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.097221    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:53:03.219502 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.219457    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42295e3-7338-4654-ae73-c08cfa9a4875-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42295e3-7338-4654-ae73-c08cfa9a4875-plugins-config-volume\") pod \"e42295e3-7338-4654-ae73-c08cfa9a4875\" (UID: \"e42295e3-7338-4654-ae73-c08cfa9a4875\") "
Oct 30 21:53:03.219707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.219605    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42295e3-7338-4654-ae73-c08cfa9a4875-kube-api-access-j2jwd\" (UniqueName: \"kubernetes.io/projected/e42295e3-7338-4654-ae73-c08cfa9a4875-kube-api-access-j2jwd\") pod \"e42295e3-7338-4654-ae73-c08cfa9a4875\" (UID: \"e42295e3-7338-4654-ae73-c08cfa9a4875\") "
Oct 30 21:53:03.219871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.219848    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42295e3-7338-4654-ae73-c08cfa9a4875-plugins-config-volume" pod "e42295e3-7338-4654-ae73-c08cfa9a4875" (UID: "e42295e3-7338-4654-ae73-c08cfa9a4875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:03.220507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.220488    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e42295e3-7338-4654-ae73-c08cfa9a4875-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:03.222060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.222032    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42295e3-7338-4654-ae73-c08cfa9a4875-kube-api-access-j2jwd" pod "e42295e3-7338-4654-ae73-c08cfa9a4875" (UID: "e42295e3-7338-4654-ae73-c08cfa9a4875"). InnerVolumeSpecName "kube-api-access-j2jwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:03.322820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.322716    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jwd\" (UniqueName: \"kubernetes.io/projected/e42295e3-7338-4654-ae73-c08cfa9a4875-kube-api-access-j2jwd\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:03.483070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.483030    4633 generic.go:356] "Generic (PLEG): container finished" podID="e42295e3-7338-4654-ae73-c08cfa9a4875" containerID="7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1" exitCode=0
Oct 30 21:53:03.483255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.483093    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"
Oct 30 21:53:03.483255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.483118    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt" event={"ID":"e42295e3-7338-4654-ae73-c08cfa9a4875","Type":"ContainerDied","Data":"7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1"}
Oct 30 21:53:03.483255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.483198    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt" event={"ID":"e42295e3-7338-4654-ae73-c08cfa9a4875","Type":"ContainerDied","Data":"019a704fb16057d37a5b9e0d470ccccdf1b0d94266eb69050db777f800965dfe"}
Oct 30 21:53:03.483255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.483221    4633 scope.go:122] "RemoveContainer" containerID="7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1"
Oct 30 21:53:03.495797 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.495778    4633 scope.go:122] "RemoveContainer" containerID="7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1"
Oct 30 21:53:03.496102 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:03.496077    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1\": container with ID starting with 7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1 not found: ID does not exist" containerID="7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1"
Oct 30 21:53:03.496173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.496115    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1"} err="failed to get container status \"7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1\": rpc error: code = NotFound desc = could not find container \"7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1\": container with ID starting with 7133f6a0fbef3ddcee3b3430245add8c7ff1267062d0e7cd06d45bf3c5031fd1 not found: ID does not exist"
Oct 30 21:53:03.512626 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.512589    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"]
Oct 30 21:53:03.517611 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.517590    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8431de-5645f848d7-dxsvt"]
Oct 30 21:53:03.793270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:03.793239    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42295e3-7338-4654-ae73-c08cfa9a4875" path="/var/lib/kubelet/pods/e42295e3-7338-4654-ae73-c08cfa9a4875/volumes"
Oct 30 21:53:05.279020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:05.278969    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"]
Oct 30 21:53:05.279570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:05.279269    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy" containerID="cri-o://e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339" gracePeriod=30
Oct 30 21:53:08.944530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:08.944489    4633 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:53:08.944998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:08.944566    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:53:10.585265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.585238    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:53:10.608843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.608814    4633 generic.go:356] "Generic (PLEG): container finished" podID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerID="e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339" exitCode=0
Oct 30 21:53:10.608996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.608863    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"
Oct 30 21:53:10.608996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.608880    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" event={"ID":"a4294be0-dd4b-4442-8f5e-9644df33121c","Type":"ContainerDied","Data":"e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339"}
Oct 30 21:53:10.608996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.608935    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv" event={"ID":"a4294be0-dd4b-4442-8f5e-9644df33121c","Type":"ContainerDied","Data":"325d962841a1b502e1bad7b68c5a2f4236dcaa29adc9a975f3a5caae4e686585"}
Oct 30 21:53:10.608996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.608966    4633 scope.go:122] "RemoveContainer" containerID="e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339"
Oct 30 21:53:10.620274 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.620244    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4294be0-dd4b-4442-8f5e-9644df33121c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4294be0-dd4b-4442-8f5e-9644df33121c-istiod-ca-cert\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.620624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.620596    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-credential-socket\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.620816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.620714    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-podinfo\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.621023 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.620985    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-credential-socket" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.621130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621111    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4294be0-dd4b-4442-8f5e-9644df33121c-istiod-ca-cert" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:10.621238 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621216    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-socket\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.621338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621264    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-token\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-token\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.621338 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621328    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-data\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.621438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621369    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-certs\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.621438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621414    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-envoy\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.621533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621466    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-kube-api-access-6xxkz\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-kube-api-access-6xxkz\") pod \"a4294be0-dd4b-4442-8f5e-9644df33121c\" (UID: \"a4294be0-dd4b-4442-8f5e-9644df33121c\") "
Oct 30 21:53:10.621662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621579    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-socket" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.621824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621614    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-data" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.621903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.621833    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-certs" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.622973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.622944    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.623129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.622978    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.623129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.622992    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.623129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.623005    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.623129 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.623021    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4294be0-dd4b-4442-8f5e-9644df33121c-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.625791 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.625764    4633 scope.go:122] "RemoveContainer" containerID="07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385"
Oct 30 21:53:10.627458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.627427    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-token" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:10.627593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.627553    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-kube-api-access-6xxkz" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "kube-api-access-6xxkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:10.627799 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.627753    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-podinfo" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:53:10.627944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.627927    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-envoy" pod "a4294be0-dd4b-4442-8f5e-9644df33121c" (UID: "a4294be0-dd4b-4442-8f5e-9644df33121c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.670643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.670615    4633 scope.go:122] "RemoveContainer" containerID="e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339"
Oct 30 21:53:10.671076 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:10.671048    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339\": container with ID starting with e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339 not found: ID does not exist" containerID="e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339"
Oct 30 21:53:10.671186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.671084    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339"} err="failed to get container status \"e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339\": rpc error: code = NotFound desc = could not find container \"e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339\": container with ID starting with e832a78f640f1e21942bc8548a1953a7ce7574b47889ba0718394342f45d7339 not found: ID does not exist"
Oct 30 21:53:10.671186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.671103    4633 scope.go:122] "RemoveContainer" containerID="07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385"
Oct 30 21:53:10.671395 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:10.671372    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385\": container with ID starting with 07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385 not found: ID does not exist" containerID="07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385"
Oct 30 21:53:10.671444 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.671403    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385"} err="failed to get container status \"07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385\": rpc error: code = NotFound desc = could not find container \"07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385\": container with ID starting with 07fa824179044b0c9f4bc316e4e5e4166953c5d264ad81d4b2c40e457c6aa385 not found: ID does not exist"
Oct 30 21:53:10.777883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.777849    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.777883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.777885    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.778141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.777902    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4294be0-dd4b-4442-8f5e-9644df33121c-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.778141 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.777917    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xxkz\" (UniqueName: \"kubernetes.io/projected/a4294be0-dd4b-4442-8f5e-9644df33121c-kube-api-access-6xxkz\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.936157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.936119    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"]
Oct 30 21:53:10.941142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:10.941112    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-mpngv"]
Oct 30 21:53:11.794933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:11.794901    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" path="/var/lib/kubelet/pods/a4294be0-dd4b-4442-8f5e-9644df33121c/volumes"
Oct 30 21:53:13.609187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:13.609089    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk" podUID="ba5084b7-93d3-4851-92fd-af91e4ed7852" containerName="main" containerID="cri-o://62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65" gracePeriod=45
Oct 30 21:53:13.944443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:13.944410    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:53:14.044880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.044844    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-dshm\") pod \"ba5084b7-93d3-4851-92fd-af91e4ed7852\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") "
Oct 30 21:53:14.045089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.045073    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-home\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-home\") pod \"ba5084b7-93d3-4851-92fd-af91e4ed7852\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") "
Oct 30 21:53:14.045174 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.045127    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-tmp-dir\") pod \"ba5084b7-93d3-4851-92fd-af91e4ed7852\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") "
Oct 30 21:53:14.045254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.045212    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-model-cache\") pod \"ba5084b7-93d3-4851-92fd-af91e4ed7852\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") "
Oct 30 21:53:14.045408 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.045386    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba5084b7-93d3-4851-92fd-af91e4ed7852-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5084b7-93d3-4851-92fd-af91e4ed7852-tls-certs\") pod \"ba5084b7-93d3-4851-92fd-af91e4ed7852\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") "
Oct 30 21:53:14.045538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.045397    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-home" pod "ba5084b7-93d3-4851-92fd-af91e4ed7852" (UID: "ba5084b7-93d3-4851-92fd-af91e4ed7852"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:14.045538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.045426    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-tmp-dir" pod "ba5084b7-93d3-4851-92fd-af91e4ed7852" (UID: "ba5084b7-93d3-4851-92fd-af91e4ed7852"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:14.045538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.045482    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5084b7-93d3-4851-92fd-af91e4ed7852-kube-api-access-dq65s\" (UniqueName: \"kubernetes.io/projected/ba5084b7-93d3-4851-92fd-af91e4ed7852-kube-api-access-dq65s\") pod \"ba5084b7-93d3-4851-92fd-af91e4ed7852\" (UID: \"ba5084b7-93d3-4851-92fd-af91e4ed7852\") "
Oct 30 21:53:14.045725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.045615    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-model-cache" pod "ba5084b7-93d3-4851-92fd-af91e4ed7852" (UID: "ba5084b7-93d3-4851-92fd-af91e4ed7852"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:14.046316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.046224    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:14.046316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.046253    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:14.046316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.046269    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:14.047711 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.047679    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-dshm" pod "ba5084b7-93d3-4851-92fd-af91e4ed7852" (UID: "ba5084b7-93d3-4851-92fd-af91e4ed7852"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:14.047833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.047747    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5084b7-93d3-4851-92fd-af91e4ed7852-kube-api-access-dq65s" pod "ba5084b7-93d3-4851-92fd-af91e4ed7852" (UID: "ba5084b7-93d3-4851-92fd-af91e4ed7852"). InnerVolumeSpecName "kube-api-access-dq65s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:14.047833 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.047798    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5084b7-93d3-4851-92fd-af91e4ed7852-tls-certs" pod "ba5084b7-93d3-4851-92fd-af91e4ed7852" (UID: "ba5084b7-93d3-4851-92fd-af91e4ed7852"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:14.147740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.147658    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba5084b7-93d3-4851-92fd-af91e4ed7852-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:14.147740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.147734    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba5084b7-93d3-4851-92fd-af91e4ed7852-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:14.147740 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.147745    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq65s\" (UniqueName: \"kubernetes.io/projected/ba5084b7-93d3-4851-92fd-af91e4ed7852-kube-api-access-dq65s\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:14.646044 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.646010    4633 generic.go:356] "Generic (PLEG): container finished" podID="ba5084b7-93d3-4851-92fd-af91e4ed7852" containerID="62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65" exitCode=0
Oct 30 21:53:14.646481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.646056    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"
Oct 30 21:53:14.646481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.646090    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk" event={"ID":"ba5084b7-93d3-4851-92fd-af91e4ed7852","Type":"ContainerDied","Data":"62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65"}
Oct 30 21:53:14.646481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.646132    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk" event={"ID":"ba5084b7-93d3-4851-92fd-af91e4ed7852","Type":"ContainerDied","Data":"417dd79c9c09cd6938bedd7fad31ce1668756080224e280606d6ba79ca5bedaa"}
Oct 30 21:53:14.646481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.646153    4633 scope.go:122] "RemoveContainer" containerID="62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65"
Oct 30 21:53:14.659169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.659148    4633 scope.go:122] "RemoveContainer" containerID="62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65"
Oct 30 21:53:14.659491 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:14.659470    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65\": container with ID starting with 62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65 not found: ID does not exist" containerID="62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65"
Oct 30 21:53:14.659563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.659500    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65"} err="failed to get container status \"62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65\": rpc error: code = NotFound desc = could not find container \"62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65\": container with ID starting with 62eaa7d9d43597304aa6f0b1156f92b0cf665cfae28298bb8b45616977c03d65 not found: ID does not exist"
Oct 30 21:53:14.671552 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.671525    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"]
Oct 30 21:53:14.676518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:14.676492    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3d9d7b/test-model-3d9d7b-kserve-cd5c4b7f7-g8tbk"]
Oct 30 21:53:15.600459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:15.600412    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:53:15.798840 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:15.798807    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5084b7-93d3-4851-92fd-af91e4ed7852" path="/var/lib/kubelet/pods/ba5084b7-93d3-4851-92fd-af91e4ed7852/volumes"
Oct 30 21:53:20.244738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:20.244701    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"]
Oct 30 21:53:25.094538 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:25.094495    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:53:25.245585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.245513    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6" podUID="ba0f6281-84e4-4b7b-a7ec-2412bfb4da99" containerName="payload-processing" containerID="cri-o://5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae" gracePeriod=25
Oct 30 21:53:25.560822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.560792    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:53:25.621310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.621221    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-kube-api-access-2hswb\" (UniqueName: \"kubernetes.io/projected/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-kube-api-access-2hswb\") pod \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\" (UID: \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\") "
Oct 30 21:53:25.621310 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.621279    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-plugins-config-volume\") pod \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\" (UID: \"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99\") "
Oct 30 21:53:25.621809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.621779    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-plugins-config-volume" pod "ba0f6281-84e4-4b7b-a7ec-2412bfb4da99" (UID: "ba0f6281-84e4-4b7b-a7ec-2412bfb4da99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:25.622195 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.622168    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:25.623828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.623802    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-kube-api-access-2hswb" pod "ba0f6281-84e4-4b7b-a7ec-2412bfb4da99" (UID: "ba0f6281-84e4-4b7b-a7ec-2412bfb4da99"). InnerVolumeSpecName "kube-api-access-2hswb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:25.726650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.726583    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hswb\" (UniqueName: \"kubernetes.io/projected/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99-kube-api-access-2hswb\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:25.857501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.857464    4633 generic.go:356] "Generic (PLEG): container finished" podID="ba0f6281-84e4-4b7b-a7ec-2412bfb4da99" containerID="5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae" exitCode=0
Oct 30 21:53:25.857731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.857512    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"
Oct 30 21:53:25.857731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.857542    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6" event={"ID":"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99","Type":"ContainerDied","Data":"5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae"}
Oct 30 21:53:25.857731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.857606    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6" event={"ID":"ba0f6281-84e4-4b7b-a7ec-2412bfb4da99","Type":"ContainerDied","Data":"c18be5d5c910f1b084b9ca8add36314743c958d1919d02e947118fbd1a3e1acb"}
Oct 30 21:53:25.857731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.857635    4633 scope.go:122] "RemoveContainer" containerID="5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae"
Oct 30 21:53:25.870113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.870089    4633 scope.go:122] "RemoveContainer" containerID="5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae"
Oct 30 21:53:25.870447 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:25.870426    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae\": container with ID starting with 5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae not found: ID does not exist" containerID="5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae"
Oct 30 21:53:25.870511 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.870456    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae"} err="failed to get container status \"5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae\": rpc error: code = NotFound desc = could not find container \"5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae\": container with ID starting with 5314b0bf61192e48165787c7ce7ef5287d8dc4a499a5cbaba5d0b3244b03e7ae not found: ID does not exist"
Oct 30 21:53:25.879241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.879174    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"]
Oct 30 21:53:25.885065 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:25.885040    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d9d7b-848d5cd79-7bkc6"]
Oct 30 21:53:27.794187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:27.794146    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0f6281-84e4-4b7b-a7ec-2412bfb4da99" path="/var/lib/kubelet/pods/ba0f6281-84e4-4b7b-a7ec-2412bfb4da99/volumes"
Oct 30 21:53:28.358348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:28.358312    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"]
Oct 30 21:53:37.573759 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:37.573722    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"]
Oct 30 21:53:37.574273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:37.573963    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy" containerID="cri-o://e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162" gracePeriod=30
Oct 30 21:53:37.582194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:37.582166    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"]
Oct 30 21:53:37.582369 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:37.582349    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy" containerID="cri-o://e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399" gracePeriod=30
Oct 30 21:53:39.537460 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:39.537419    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"]
Oct 30 21:53:42.927818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:42.927789    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:53:42.979790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:42.979751    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:53:43.021271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021235    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-envoy\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.021473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021292    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-certs\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.021473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021344    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istiod-ca-cert\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.021688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021484    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-data\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.021688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021620    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-podinfo\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.021811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021784    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-socket\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.021871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021827    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-kube-api-access-8zzg2\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-kube-api-access-8zzg2\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.021939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021857    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-certs" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.021939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021874    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-token\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-token\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.021939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.021905    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-credential-socket\") pod \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\" (UID: \"f9785510-14ac-4cb6-bde5-b15c4cf827d4\") "
Oct 30 21:53:43.023039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.023008    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-credential-socket" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.023293 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.023266    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.023388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.023341    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istiod-ca-cert" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:43.023514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.023489    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-socket" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.023585 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.023569    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-data" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.024942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.024912    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-envoy" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.025865 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.025829    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-podinfo" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:53:43.026307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.026283    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-kube-api-access-8zzg2" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "kube-api-access-8zzg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:43.027184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.027160    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-token" pod "f9785510-14ac-4cb6-bde5-b15c4cf827d4" (UID: "f9785510-14ac-4cb6-bde5-b15c4cf827d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:43.124864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.124825    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-data\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.125075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.124884    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df70c75b-9dcd-44e9-8a6d-0195be290365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df70c75b-9dcd-44e9-8a6d-0195be290365-istiod-ca-cert\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.125075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.125044    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-certs\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.125217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.125115    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-kube-api-access-7rvvj\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-kube-api-access-7rvvj\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.125287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.125242    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-podinfo\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.125287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.125278    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-envoy\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.125544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.125239    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-data" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.125544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.125335    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-token\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-token\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.125544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.125410    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-certs" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.125544 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.125421    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df70c75b-9dcd-44e9-8a6d-0195be290365-istiod-ca-cert" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:43.126164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.126134    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-credential-socket\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.126346 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.126227    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-socket\") pod \"df70c75b-9dcd-44e9-8a6d-0195be290365\" (UID: \"df70c75b-9dcd-44e9-8a6d-0195be290365\") "
Oct 30 21:53:43.126512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.126410    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-credential-socket" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.126712 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.126641    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-socket" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.127290 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127263    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.127407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127296    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.127407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127311    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.127407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127327    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.127407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127341    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.127407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127355    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.127407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127370    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.127407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127382    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.127407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127396    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df70c75b-9dcd-44e9-8a6d-0195be290365-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.128016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127413    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zzg2\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-kube-api-access-8zzg2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.128016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127427    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9785510-14ac-4cb6-bde5-b15c4cf827d4-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.128016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127441    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9785510-14ac-4cb6-bde5-b15c4cf827d4-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.128016 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.127456    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.128247 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.128150    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-envoy" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.128832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.128616    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-token" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:43.128832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.128786    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-kube-api-access-7rvvj" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "kube-api-access-7rvvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:43.129614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.129558    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-podinfo" pod "df70c75b-9dcd-44e9-8a6d-0195be290365" (UID: "df70c75b-9dcd-44e9-8a6d-0195be290365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:53:43.229294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.229256    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rvvj\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-kube-api-access-7rvvj\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.229294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.229291    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.229517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.229307    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.229517 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.229319    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df70c75b-9dcd-44e9-8a6d-0195be290365-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.268268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.268226    4633 generic.go:356] "Generic (PLEG): container finished" podID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerID="e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162" exitCode=0
Oct 30 21:53:43.268459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.268274    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"
Oct 30 21:53:43.268459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.268311    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" event={"ID":"f9785510-14ac-4cb6-bde5-b15c4cf827d4","Type":"ContainerDied","Data":"e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162"}
Oct 30 21:53:43.268459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.268369    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht" event={"ID":"f9785510-14ac-4cb6-bde5-b15c4cf827d4","Type":"ContainerDied","Data":"2bcf6d109a5a183d653222a6e0f10d8feabefebd2b75e462eb9834b581c3af28"}
Oct 30 21:53:43.268459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.268399    4633 scope.go:122] "RemoveContainer" containerID="e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162"
Oct 30 21:53:43.270113 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.270087    4633 generic.go:356] "Generic (PLEG): container finished" podID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerID="e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399" exitCode=0
Oct 30 21:53:43.270209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.270128    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"
Oct 30 21:53:43.270209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.270176    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" event={"ID":"df70c75b-9dcd-44e9-8a6d-0195be290365","Type":"ContainerDied","Data":"e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399"}
Oct 30 21:53:43.270313 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.270220    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb" event={"ID":"df70c75b-9dcd-44e9-8a6d-0195be290365","Type":"ContainerDied","Data":"3ba4d9cff7d0caa208f6459b68aca36c9e3cef9c15b5ba00568196d064344450"}
Oct 30 21:53:43.282643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.282621    4633 scope.go:122] "RemoveContainer" containerID="aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052"
Oct 30 21:53:43.294564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.294539    4633 scope.go:122] "RemoveContainer" containerID="e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162"
Oct 30 21:53:43.294942 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:43.294915    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162\": container with ID starting with e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162 not found: ID does not exist" containerID="e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162"
Oct 30 21:53:43.295031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.294951    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162"} err="failed to get container status \"e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162\": rpc error: code = NotFound desc = could not find container \"e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162\": container with ID starting with e3bdbc3dbb7b96b99e0dad480f1d6b5f185044e5daa485d39b470600c20da162 not found: ID does not exist"
Oct 30 21:53:43.295031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.294973    4633 scope.go:122] "RemoveContainer" containerID="aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052"
Oct 30 21:53:43.295267 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:43.295247    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052\": container with ID starting with aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052 not found: ID does not exist" containerID="aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052"
Oct 30 21:53:43.295315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.295273    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052"} err="failed to get container status \"aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052\": rpc error: code = NotFound desc = could not find container \"aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052\": container with ID starting with aed374a572a366cc2611d341ba6596a29f8513b1346d5be6df3f3c59d7d62052 not found: ID does not exist"
Oct 30 21:53:43.295315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.295289    4633 scope.go:122] "RemoveContainer" containerID="e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399"
Oct 30 21:53:43.302469 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.302443    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"]
Oct 30 21:53:43.310026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.309999    4633 scope.go:122] "RemoveContainer" containerID="e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399"
Oct 30 21:53:43.310441 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:43.310418    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399\": container with ID starting with e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399 not found: ID does not exist" containerID="e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399"
Oct 30 21:53:43.310527 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.310452    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399"} err="failed to get container status \"e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399\": rpc error: code = NotFound desc = could not find container \"e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399\": container with ID starting with e5a6663aee13c99c724e848503380548ae32b31ce8b90eb8bcb4bff1b4906399 not found: ID does not exist"
Oct 30 21:53:43.310942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.310924    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-tqxwb"]
Oct 30 21:53:43.325699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.325653    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"]
Oct 30 21:53:43.331569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.331539    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-570f01-openshift-default-5c4c4995f5-mnzht"]
Oct 30 21:53:43.384001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.383911    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f" podUID="f8023d07-bf60-418f-b26c-1ef210b5f032" containerName="main" containerID="cri-o://a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605" gracePeriod=45
Oct 30 21:53:43.759374 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.759344    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:53:43.797986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.797944    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" path="/var/lib/kubelet/pods/df70c75b-9dcd-44e9-8a6d-0195be290365/volumes"
Oct 30 21:53:43.798835 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.798810    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" path="/var/lib/kubelet/pods/f9785510-14ac-4cb6-bde5-b15c4cf827d4/volumes"
Oct 30 21:53:43.845873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.845839    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-tmp-dir\") pod \"f8023d07-bf60-418f-b26c-1ef210b5f032\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") "
Oct 30 21:53:43.845873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.845880    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8023d07-bf60-418f-b26c-1ef210b5f032-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8023d07-bf60-418f-b26c-1ef210b5f032-tls-certs\") pod \"f8023d07-bf60-418f-b26c-1ef210b5f032\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") "
Oct 30 21:53:43.846109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.846039    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-model-cache\") pod \"f8023d07-bf60-418f-b26c-1ef210b5f032\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") "
Oct 30 21:53:43.846203 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.846174    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-tmp-dir" pod "f8023d07-bf60-418f-b26c-1ef210b5f032" (UID: "f8023d07-bf60-418f-b26c-1ef210b5f032"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.846281 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.846263    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-dshm\") pod \"f8023d07-bf60-418f-b26c-1ef210b5f032\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") "
Oct 30 21:53:43.846341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.846268    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-model-cache" pod "f8023d07-bf60-418f-b26c-1ef210b5f032" (UID: "f8023d07-bf60-418f-b26c-1ef210b5f032"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.846390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.846355    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8023d07-bf60-418f-b26c-1ef210b5f032-kube-api-access-5pf8x\" (UniqueName: \"kubernetes.io/projected/f8023d07-bf60-418f-b26c-1ef210b5f032-kube-api-access-5pf8x\") pod \"f8023d07-bf60-418f-b26c-1ef210b5f032\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") "
Oct 30 21:53:43.846442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.846403    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-home\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-home\") pod \"f8023d07-bf60-418f-b26c-1ef210b5f032\" (UID: \"f8023d07-bf60-418f-b26c-1ef210b5f032\") "
Oct 30 21:53:43.846738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.846712    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-home" pod "f8023d07-bf60-418f-b26c-1ef210b5f032" (UID: "f8023d07-bf60-418f-b26c-1ef210b5f032"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.847159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.847135    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.847159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.847160    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.847322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.847173    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.848909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.848888    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8023d07-bf60-418f-b26c-1ef210b5f032-kube-api-access-5pf8x" pod "f8023d07-bf60-418f-b26c-1ef210b5f032" (UID: "f8023d07-bf60-418f-b26c-1ef210b5f032"). InnerVolumeSpecName "kube-api-access-5pf8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:43.848998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.848981    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-dshm" pod "f8023d07-bf60-418f-b26c-1ef210b5f032" (UID: "f8023d07-bf60-418f-b26c-1ef210b5f032"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:43.849043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.848985    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8023d07-bf60-418f-b26c-1ef210b5f032-tls-certs" pod "f8023d07-bf60-418f-b26c-1ef210b5f032" (UID: "f8023d07-bf60-418f-b26c-1ef210b5f032"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:43.948427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.948391    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8023d07-bf60-418f-b26c-1ef210b5f032-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.948427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.948426    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pf8x\" (UniqueName: \"kubernetes.io/projected/f8023d07-bf60-418f-b26c-1ef210b5f032-kube-api-access-5pf8x\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.948941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:43.948442    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8023d07-bf60-418f-b26c-1ef210b5f032-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:44.283256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.283221    4633 generic.go:356] "Generic (PLEG): container finished" podID="f8023d07-bf60-418f-b26c-1ef210b5f032" containerID="a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605" exitCode=0
Oct 30 21:53:44.283481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.283265    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f" event={"ID":"f8023d07-bf60-418f-b26c-1ef210b5f032","Type":"ContainerDied","Data":"a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605"}
Oct 30 21:53:44.283481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.283275    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"
Oct 30 21:53:44.283481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.283304    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f" event={"ID":"f8023d07-bf60-418f-b26c-1ef210b5f032","Type":"ContainerDied","Data":"663133f5efe307b09a088c6f4b49f421e50bf7314c8ff1f9ccb2f8d7900d2b06"}
Oct 30 21:53:44.283481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.283324    4633 scope.go:122] "RemoveContainer" containerID="a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605"
Oct 30 21:53:44.296757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.296733    4633 scope.go:122] "RemoveContainer" containerID="a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605"
Oct 30 21:53:44.297090 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:44.297070    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605\": container with ID starting with a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605 not found: ID does not exist" containerID="a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605"
Oct 30 21:53:44.297157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.297100    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605"} err="failed to get container status \"a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605\": rpc error: code = NotFound desc = could not find container \"a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605\": container with ID starting with a3b8935fa62bf4a835109807e1d42264cd65c710f5d4df1c8c4144ce0bf26605 not found: ID does not exist"
Oct 30 21:53:44.309633 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.309599    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"]
Oct 30 21:53:44.314710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.314635    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f"]
Oct 30 21:53:44.689696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:44.688757    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-cf51d4/test-model-cf51d4-kserve-8df4bc6d9-h9f4f" podUID="f8023d07-bf60-418f-b26c-1ef210b5f032" containerName="main" probeResult="failure" output="Get \"https://10.133.1.9:8000/ready\": context deadline exceeded"
Oct 30 21:53:45.794899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:45.794864    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8023d07-bf60-418f-b26c-1ef210b5f032" path="/var/lib/kubelet/pods/f8023d07-bf60-418f-b26c-1ef210b5f032/volumes"
Oct 30 21:53:54.581696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.573606    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh" podUID="b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" containerName="main" containerID="cri-o://21e4bb58c001f6a69869d6993617587cd38be488651a5f9783e05c0afa505638" gracePeriod=45
Oct 30 21:53:54.840425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.840332    4633 generic.go:356] "Generic (PLEG): container finished" podID="b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" containerID="21e4bb58c001f6a69869d6993617587cd38be488651a5f9783e05c0afa505638" exitCode=0
Oct 30 21:53:54.840425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.840369    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh" event={"ID":"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb","Type":"ContainerDied","Data":"21e4bb58c001f6a69869d6993617587cd38be488651a5f9783e05c0afa505638"}
Oct 30 21:53:54.904177 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.904148    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:53:54.996473 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.996435    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-dshm\") pod \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") "
Oct 30 21:53:54.996729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.996520    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-kube-api-access-wwbn5\" (UniqueName: \"kubernetes.io/projected/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-kube-api-access-wwbn5\") pod \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") "
Oct 30 21:53:54.996729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.996627    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tls-certs\") pod \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") "
Oct 30 21:53:54.996860 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.996767    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-model-cache\") pod \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") "
Oct 30 21:53:54.996919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.996900    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-home\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-home\") pod \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") "
Oct 30 21:53:54.996996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.996972    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tmp-dir\") pod \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\" (UID: \"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb\") "
Oct 30 21:53:54.997330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.997276    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-model-cache" pod "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" (UID: "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:54.997330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.997296    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-home" pod "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" (UID: "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:54.997507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.997412    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tmp-dir" pod "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" (UID: "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:54.998024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.997998    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:54.998125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.998030    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:54.998125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.998046    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:54.999816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.999795    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-dshm" pod "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" (UID: "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:54.999914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:54.999858    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tls-certs" pod "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" (UID: "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:55.002863 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.002837    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-kube-api-access-wwbn5" pod "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" (UID: "b1aa404d-1c9e-47ec-b67d-e3f8124a17eb"). InnerVolumeSpecName "kube-api-access-wwbn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:55.022496 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:53:55.022463    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:53:55.099181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.099098    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:55.099181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.099126    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwbn5\" (UniqueName: \"kubernetes.io/projected/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-kube-api-access-wwbn5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:55.099181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.099136    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:55.849921 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.849888    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"
Oct 30 21:53:55.850394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.849924    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh" event={"ID":"b1aa404d-1c9e-47ec-b67d-e3f8124a17eb","Type":"ContainerDied","Data":"420b71bf6c4f9063c284006acfc42990627b1ef4cf36abefc68ec33903cdd06a"}
Oct 30 21:53:55.850394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.849982    4633 scope.go:122] "RemoveContainer" containerID="21e4bb58c001f6a69869d6993617587cd38be488651a5f9783e05c0afa505638"
Oct 30 21:53:55.875995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.875957    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"]
Oct 30 21:53:55.881173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:55.881147    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-87ae68/test-model-87ae68-kserve-8bdd945bf-z4lgh"]
Oct 30 21:53:57.793412 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:57.793373    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" path="/var/lib/kubelet/pods/b1aa404d-1c9e-47ec-b67d-e3f8124a17eb/volumes"
Oct 30 21:53:59.471163 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:59.471126    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"]
Oct 30 21:53:59.471660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:59.471374    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy" containerID="cri-o://d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495" gracePeriod=30
Oct 30 21:53:59.481685 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:59.481643    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"]
Oct 30 21:53:59.481943 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:59.481916    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" containerID="cri-o://e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165" gracePeriod=30
Oct 30 21:53:59.488146 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:59.488117    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"]
Oct 30 21:53:59.488380 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:59.488341    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" containerID="cri-o://ade44335502cab19cd3da79450c2e7f19f8e417cbfd840dd6ac336eb66a109a2" gracePeriod=30
Oct 30 21:53:59.493222 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:59.493197    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"]
Oct 30 21:53:59.493416 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:53:59.493395    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" containerID="cri-o://1a7effacddc4f3cc73689015d0f788f5170118c25c7700fff7a474cba8bc19ea" gracePeriod=30
Oct 30 21:54:00.332487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:00.332446    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:00.332659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:00.332533    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:00.414427 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:00.414391    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:00.414662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:00.414469    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:01.196178 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:01.196135    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:01.196558 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:01.196216    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:01.225936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:01.225903    4633 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:01.226102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:01.225956    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:04.838422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.837413    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:54:04.851770 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.851704    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:54:04.945378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.945337    4633 generic.go:356] "Generic (PLEG): container finished" podID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerID="e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165" exitCode=0
Oct 30 21:54:04.945540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.945493    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" event={"ID":"e6c14236-fd6a-404b-8219-5ff6ba2f98f3","Type":"ContainerDied","Data":"e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165"}
Oct 30 21:54:04.945734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.945549    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb" event={"ID":"e6c14236-fd6a-404b-8219-5ff6ba2f98f3","Type":"ContainerDied","Data":"f8d6f007e2de60cb35edb8fb3da9c1d567e65c731124b0b4df61e066cb7999a8"}
Oct 30 21:54:04.945734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.945579    4633 scope.go:122] "RemoveContainer" containerID="e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165"
Oct 30 21:54:04.945900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.945753    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"
Oct 30 21:54:04.947964 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.947940    4633 generic.go:356] "Generic (PLEG): container finished" podID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerID="d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495" exitCode=0
Oct 30 21:54:04.948104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.947983    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"
Oct 30 21:54:04.948104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.948024    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" event={"ID":"6f45d142-8150-4eec-9dbb-58e94d3d171d","Type":"ContainerDied","Data":"d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495"}
Oct 30 21:54:04.948104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.948070    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx" event={"ID":"6f45d142-8150-4eec-9dbb-58e94d3d171d","Type":"ContainerDied","Data":"382adbd31b2fd5cf2eaa473ea31db06d8ce62a000bb463112a714888f04e532f"}
Oct 30 21:54:04.949765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.949735    4633 generic.go:356] "Generic (PLEG): container finished" podID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerID="ade44335502cab19cd3da79450c2e7f19f8e417cbfd840dd6ac336eb66a109a2" exitCode=0
Oct 30 21:54:04.949899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.949826    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" event={"ID":"244ce12a-d4e5-4ee7-9b57-14b1f13f345b","Type":"ContainerDied","Data":"ade44335502cab19cd3da79450c2e7f19f8e417cbfd840dd6ac336eb66a109a2"}
Oct 30 21:54:04.951569 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.951549    4633 generic.go:356] "Generic (PLEG): container finished" podID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerID="1a7effacddc4f3cc73689015d0f788f5170118c25c7700fff7a474cba8bc19ea" exitCode=0
Oct 30 21:54:04.951704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.951609    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" event={"ID":"a521156d-a012-4fe7-8873-a30d24fa3bdd","Type":"ContainerDied","Data":"1a7effacddc4f3cc73689015d0f788f5170118c25c7700fff7a474cba8bc19ea"}
Oct 30 21:54:04.956117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.956097    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:54:04.957068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957042    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-envoy\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.957228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957211    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-podinfo\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.957307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957263    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f45d142-8150-4eec-9dbb-58e94d3d171d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45d142-8150-4eec-9dbb-58e94d3d171d-istiod-ca-cert\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.957376 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957311    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-envoy\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.957435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957393    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-credential-socket\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.957499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957441    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-socket\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.957499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957479    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-credential-socket\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.957614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957509    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-socket\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.957614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957568    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-kube-api-access-jd525\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-kube-api-access-jd525\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.957614 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957605    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-podinfo\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.957801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957645    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-kube-api-access-pf7zs\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-kube-api-access-pf7zs\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.957801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957711    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-data\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.957801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957780    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f45d142-8150-4eec-9dbb-58e94d3d171d-istiod-ca-cert" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:04.958008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957801    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-certs\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.958008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957843    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-token\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-token\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.958008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957936    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-certs\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.958008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957957    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-socket" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.958008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957963    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-socket" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.958008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.957978    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istiod-ca-cert\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.958349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.958031    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-token\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-token\") pod \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\" (UID: \"e6c14236-fd6a-404b-8219-5ff6ba2f98f3\") "
Oct 30 21:54:04.958349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.958056    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-data\") pod \"6f45d142-8150-4eec-9dbb-58e94d3d171d\" (UID: \"6f45d142-8150-4eec-9dbb-58e94d3d171d\") "
Oct 30 21:54:04.958349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.958121    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-credential-socket" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.958349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.958125    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-credential-socket" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.958919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.958848    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istiod-ca-cert" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:04.959025 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959000    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f45d142-8150-4eec-9dbb-58e94d3d171d-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.959123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959032    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.959123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959047    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.959123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959061    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.959123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959075    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.959269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959140    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-data" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.959269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959173    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-certs" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.959783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959754    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-data" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.960019 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.959996    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-certs" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.961792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.961763    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-token" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:04.961899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.961808    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-token" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:04.963132 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.963104    4633 scope.go:122] "RemoveContainer" containerID="e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165"
Oct 30 21:54:04.963339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.963320    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-podinfo" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:04.963441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.963374    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-kube-api-access-jd525" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "kube-api-access-jd525". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:04.963441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.963390    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-podinfo" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:04.963441 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:04.963407    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165\": container with ID starting with e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165 not found: ID does not exist" containerID="e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165"
Oct 30 21:54:04.963441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.963437    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-envoy" pod "6f45d142-8150-4eec-9dbb-58e94d3d171d" (UID: "6f45d142-8150-4eec-9dbb-58e94d3d171d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.963880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.963440    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165"} err="failed to get container status \"e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165\": rpc error: code = NotFound desc = could not find container \"e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165\": container with ID starting with e59b693814f57a9434c63cb3ad0eb4def5097c1e12d0a555626035da00f12165 not found: ID does not exist"
Oct 30 21:54:04.963880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.963465    4633 scope.go:122] "RemoveContainer" containerID="d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495"
Oct 30 21:54:04.963880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.963781    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:54:04.964499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.964476    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-envoy" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.965353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.965335    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-kube-api-access-pf7zs" pod "e6c14236-fd6a-404b-8219-5ff6ba2f98f3" (UID: "e6c14236-fd6a-404b-8219-5ff6ba2f98f3"). InnerVolumeSpecName "kube-api-access-pf7zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:04.975455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.975434    4633 scope.go:122] "RemoveContainer" containerID="d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495"
Oct 30 21:54:04.975803 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:04.975779    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495\": container with ID starting with d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495 not found: ID does not exist" containerID="d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495"
Oct 30 21:54:04.975877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:04.975815    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495"} err="failed to get container status \"d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495\": rpc error: code = NotFound desc = could not find container \"d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495\": container with ID starting with d5175ae93e327d6f56cd247db935bc30125c47df631dc1354e25c6efbc76a495 not found: ID does not exist"
Oct 30 21:54:05.059996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.059903    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-socket\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.059996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.059962    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-token\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-token\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.059996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.059995    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-kube-api-access-vmm84\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-kube-api-access-vmm84\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.060304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060021    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-certs\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.060304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060076    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istiod-ca-cert\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.060304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060128    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-credential-socket\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.060304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060265    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-socket" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.060304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060290    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-kube-api-access-vjbdx\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-kube-api-access-vjbdx\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.060539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060321    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-data\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.060539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060364    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-certs\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.060539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060424    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-socket\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.060539 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060509    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-envoy\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.060707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060554    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a521156d-a012-4fe7-8873-a30d24fa3bdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521156d-a012-4fe7-8873-a30d24fa3bdd-istiod-ca-cert\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.060707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060568    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istiod-ca-cert" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:05.060707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060634    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-data\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.060707 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060692    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-podinfo\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.060939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060737    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-credential-socket\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.060939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060780    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-envoy\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.060939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060881    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-token\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-token\") pod \"a521156d-a012-4fe7-8873-a30d24fa3bdd\" (UID: \"a521156d-a012-4fe7-8873-a30d24fa3bdd\") "
Oct 30 21:54:05.061262 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.060946    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-podinfo\") pod \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\" (UID: \"244ce12a-d4e5-4ee7-9b57-14b1f13f345b\") "
Oct 30 21:54:05.061262 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061153    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-data" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.061883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061810    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd525\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-kube-api-access-jd525\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.061883 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061869    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061887    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf7zs\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-kube-api-access-pf7zs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061903    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061898    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-certs" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061917    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061931    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061947    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061961    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061975    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.061989    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062004    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062017    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062033    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062047    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062061    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f45d142-8150-4eec-9dbb-58e94d3d171d-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062074    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c14236-fd6a-404b-8219-5ff6ba2f98f3-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.062137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062075    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-credential-socket" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.063370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062301    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-credential-socket" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.063370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062556    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-data" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.063370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062874    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a521156d-a012-4fe7-8873-a30d24fa3bdd-istiod-ca-cert" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:05.063370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.062898    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-kube-api-access-vmm84" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "kube-api-access-vmm84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:05.063370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.063037    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-token" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:05.063370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.063161    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-socket" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.063700 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.063648    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-kube-api-access-vjbdx" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "kube-api-access-vjbdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:05.064018 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.063997    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-podinfo" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:05.064784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.064755    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-envoy" pod "244ce12a-d4e5-4ee7-9b57-14b1f13f345b" (UID: "244ce12a-d4e5-4ee7-9b57-14b1f13f345b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.064877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.064851    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-podinfo" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:05.065125 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.065104    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-envoy" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.065803 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.065784    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-certs" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:05.066442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.066424    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-token" pod "a521156d-a012-4fe7-8873-a30d24fa3bdd" (UID: "a521156d-a012-4fe7-8873-a30d24fa3bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:05.164287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164249    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164284    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmm84\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-kube-api-access-vmm84\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164298    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164312    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164324    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjbdx\" (UniqueName: \"kubernetes.io/projected/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-kube-api-access-vjbdx\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164337    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164348    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164359    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164371    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a521156d-a012-4fe7-8873-a30d24fa3bdd-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164383    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164395    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164406    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164419    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164430    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a521156d-a012-4fe7-8873-a30d24fa3bdd-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.164516 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.164442    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244ce12a-d4e5-4ee7-9b57-14b1f13f345b-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.293365 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.293332    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"]
Oct 30 21:54:05.302504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.302472    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-5c5vb"]
Oct 30 21:54:05.324180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.324101    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"]
Oct 30 21:54:05.340636 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.340607    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-lvjvx"]
Oct 30 21:54:05.793824 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.793785    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" path="/var/lib/kubelet/pods/6f45d142-8150-4eec-9dbb-58e94d3d171d/volumes"
Oct 30 21:54:05.794309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.794295    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" path="/var/lib/kubelet/pods/e6c14236-fd6a-404b-8219-5ff6ba2f98f3/volumes"
Oct 30 21:54:05.964184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.964145    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n" event={"ID":"244ce12a-d4e5-4ee7-9b57-14b1f13f345b","Type":"ContainerDied","Data":"3552e58088127aeeb451483dddb84babc44a38254f6fa28d45304759076865f2"}
Oct 30 21:54:05.964724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.964162    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"
Oct 30 21:54:05.964724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.964212    4633 scope.go:122] "RemoveContainer" containerID="ade44335502cab19cd3da79450c2e7f19f8e417cbfd840dd6ac336eb66a109a2"
Oct 30 21:54:05.965823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.965804    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb" event={"ID":"a521156d-a012-4fe7-8873-a30d24fa3bdd","Type":"ContainerDied","Data":"b880ca41cfd6b34083297c62fb3ce75730f3140cb5dc5f93334f57fb9480bca4"}
Oct 30 21:54:05.965916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.965831    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"
Oct 30 21:54:05.976660 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.976633    4633 scope.go:122] "RemoveContainer" containerID="1a7effacddc4f3cc73689015d0f788f5170118c25c7700fff7a474cba8bc19ea"
Oct 30 21:54:05.995812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.995764    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"]
Oct 30 21:54:05.999118 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:05.999091    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-b7n4n"]
Oct 30 21:54:06.015532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:06.015501    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"]
Oct 30 21:54:06.019809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:06.019786    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-mxkkb"]
Oct 30 21:54:07.796148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:07.796113    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" path="/var/lib/kubelet/pods/244ce12a-d4e5-4ee7-9b57-14b1f13f345b/volumes"
Oct 30 21:54:07.796822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:07.796654    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" path="/var/lib/kubelet/pods/a521156d-a012-4fe7-8873-a30d24fa3bdd/volumes"
Oct 30 21:54:07.947919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:07.947871    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"]
Oct 30 21:54:07.948617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:07.948565    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy" containerID="cri-o://bac570f7876d3f60c92ec9d487a1d37d69194d47fb894e108e1bb18a77d77935" gracePeriod=30
Oct 30 21:54:10.574613 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:10.574576    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"]
Oct 30 21:54:13.269747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.269421    4633 generic.go:356] "Generic (PLEG): container finished" podID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerID="bac570f7876d3f60c92ec9d487a1d37d69194d47fb894e108e1bb18a77d77935" exitCode=0
Oct 30 21:54:13.269747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.269551    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" event={"ID":"e4c30d58-399a-4f54-b1c5-96ed607bcf89","Type":"ContainerDied","Data":"bac570f7876d3f60c92ec9d487a1d37d69194d47fb894e108e1bb18a77d77935"}
Oct 30 21:54:13.491474 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.491449    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:54:13.535029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.534953    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-kube-api-access-h4pw6\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-kube-api-access-h4pw6\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.535029 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.535005    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-token\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-token\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.535319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.535080    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-certs\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.535319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.535129    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-data\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.535319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.535162    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-podinfo\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.535319 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.535258    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istiod-ca-cert\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.535582 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.535355    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-socket\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.535708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.535639    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-data" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.535922 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.535897    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-certs" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.536226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.536202    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istiod-ca-cert" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:13.536629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.536587    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-socket" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.537229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.537199    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.537349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.537235    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.537349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.537250    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.537349 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.537265    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.537913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.537889    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-kube-api-access-h4pw6" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "kube-api-access-h4pw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:13.538028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.537972    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-token" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:13.538724 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.538656    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-podinfo" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:13.638591 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.638562    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-envoy\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.638933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.638903    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-credential-socket\") pod \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\" (UID: \"e4c30d58-399a-4f54-b1c5-96ed607bcf89\") "
Oct 30 21:54:13.639160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.639139    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-credential-socket" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.639428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.639412    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.639471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.639433    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4pw6\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-kube-api-access-h4pw6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.639471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.639444    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.639471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.639453    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.640834 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.640816    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-envoy" pod "e4c30d58-399a-4f54-b1c5-96ed607bcf89" (UID: "e4c30d58-399a-4f54-b1c5-96ed607bcf89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.758562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:13.756511    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c30d58-399a-4f54-b1c5-96ed607bcf89-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.279439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:14.279404    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"
Oct 30 21:54:14.279439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:14.279429    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x" event={"ID":"e4c30d58-399a-4f54-b1c5-96ed607bcf89","Type":"ContainerDied","Data":"f30ff144a6583d0f1e824cd9c59fa4b8c4831a29e4f613130e52c0e910bc036e"}
Oct 30 21:54:14.280017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:14.279492    4633 scope.go:122] "RemoveContainer" containerID="bac570f7876d3f60c92ec9d487a1d37d69194d47fb894e108e1bb18a77d77935"
Oct 30 21:54:14.302232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:14.302202    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"]
Oct 30 21:54:14.308439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:14.308413    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-j724x"]
Oct 30 21:54:15.575054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:15.574990    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl" podUID="5b8f84c7-6449-4add-be75-f2e3819e04a7" containerName="payload-pre-processing" containerID="cri-o://a574a24bc46c51fade833df30555b55d2aac37e026ffb0327e0e25e3666d4f46" gracePeriod=25
Oct 30 21:54:15.793784 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:15.793747    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" path="/var/lib/kubelet/pods/e4c30d58-399a-4f54-b1c5-96ed607bcf89/volumes"
Oct 30 21:54:18.885584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:18.878230    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"]
Oct 30 21:54:18.945697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:18.944983    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf" podUID="30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" containerName="maas-api" containerID="cri-o://39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b" gracePeriod=30
Oct 30 21:54:19.612729 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.612653    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:54:19.721745 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.721711    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-metrics-certs\") pod \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") "
Oct 30 21:54:19.721959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.721761    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-openshift-service-ca\") pod \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") "
Oct 30 21:54:19.721959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.721783    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-kube-api-access-bwlzv\" (UniqueName: \"kubernetes.io/projected/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-kube-api-access-bwlzv\") pod \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") "
Oct 30 21:54:19.721959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.721818    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-maas-api-tls\") pod \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\" (UID: \"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e\") "
Oct 30 21:54:19.722187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.722158    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-openshift-service-ca" pod "30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" (UID: "30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:19.722356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.722337    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:19.724156 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.724123    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-maas-api-tls" pod "30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" (UID: "30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:19.724273 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.724200    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-metrics-certs" pod "30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" (UID: "30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:19.724325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.724277    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-kube-api-access-bwlzv" pod "30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" (UID: "30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e"). InnerVolumeSpecName "kube-api-access-bwlzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:19.823426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.823394    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:19.823426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.823421    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwlzv\" (UniqueName: \"kubernetes.io/projected/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-kube-api-access-bwlzv\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:19.823426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:19.823433    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:20.192244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.192206    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"]
Oct 30 21:54:20.192765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.192503    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj" podUID="656dbefb-91f4-48ce-afe3-742b52686788" containerName="maas-api" containerID="cri-o://81073fc4516b1d92a45f9cf767e99323f46df047cf42f55001df44d8787ce657" gracePeriod=30
Oct 30 21:54:20.432695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.393624    4633 generic.go:356] "Generic (PLEG): container finished" podID="30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" containerID="39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b" exitCode=0
Oct 30 21:54:20.432695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.393847    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf" event={"ID":"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e","Type":"ContainerDied","Data":"39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b"}
Oct 30 21:54:20.432695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.393901    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf" event={"ID":"30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e","Type":"ContainerDied","Data":"9d4e32cb8aeb0104187a73127468f3e47f55d67de71bd8c8cda72f6cfd2c78e5"}
Oct 30 21:54:20.432695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.393929    4633 scope.go:122] "RemoveContainer" containerID="39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b"
Oct 30 21:54:20.432695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.394105    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"
Oct 30 21:54:20.489448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.489403    4633 generic.go:356] "Generic (PLEG): container finished" podID="656dbefb-91f4-48ce-afe3-742b52686788" containerID="81073fc4516b1d92a45f9cf767e99323f46df047cf42f55001df44d8787ce657" exitCode=0
Oct 30 21:54:20.489738 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.489538    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj" event={"ID":"656dbefb-91f4-48ce-afe3-742b52686788","Type":"ContainerDied","Data":"81073fc4516b1d92a45f9cf767e99323f46df047cf42f55001df44d8787ce657"}
Oct 30 21:54:20.656036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.656004    4633 scope.go:122] "RemoveContainer" containerID="39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b"
Oct 30 21:54:20.657377 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:20.656661    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b\": container with ID starting with 39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b not found: ID does not exist" containerID="39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b"
Oct 30 21:54:20.657377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.656837    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b"} err="failed to get container status \"39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b\": rpc error: code = NotFound desc = could not find container \"39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b\": container with ID starting with 39ddfad0febc1da39bbf68b96692e4c69ba1b39b342656dd92d000093b842a3b not found: ID does not exist"
Oct 30 21:54:20.687788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.687741    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"]
Oct 30 21:54:20.689348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.689322    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-87ae68-78755fcc9b-bmbrf"]
Oct 30 21:54:20.730551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.730480    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:54:20.764789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.764758    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656dbefb-91f4-48ce-afe3-742b52686788-kube-api-access-pf9lw\" (UniqueName: \"kubernetes.io/projected/656dbefb-91f4-48ce-afe3-742b52686788-kube-api-access-pf9lw\") pod \"656dbefb-91f4-48ce-afe3-742b52686788\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") "
Oct 30 21:54:20.764789 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.764794    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-maas-api-tls\") pod \"656dbefb-91f4-48ce-afe3-742b52686788\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") "
Oct 30 21:54:20.765056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.764924    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656dbefb-91f4-48ce-afe3-742b52686788-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/656dbefb-91f4-48ce-afe3-742b52686788-openshift-service-ca\") pod \"656dbefb-91f4-48ce-afe3-742b52686788\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") "
Oct 30 21:54:20.765056 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.765032    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-metrics-certs\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-metrics-certs\") pod \"656dbefb-91f4-48ce-afe3-742b52686788\" (UID: \"656dbefb-91f4-48ce-afe3-742b52686788\") "
Oct 30 21:54:20.765284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.765255    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656dbefb-91f4-48ce-afe3-742b52686788-openshift-service-ca" pod "656dbefb-91f4-48ce-afe3-742b52686788" (UID: "656dbefb-91f4-48ce-afe3-742b52686788"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:20.766143 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.766117    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/656dbefb-91f4-48ce-afe3-742b52686788-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:20.767543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.767322    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656dbefb-91f4-48ce-afe3-742b52686788-kube-api-access-pf9lw" pod "656dbefb-91f4-48ce-afe3-742b52686788" (UID: "656dbefb-91f4-48ce-afe3-742b52686788"). InnerVolumeSpecName "kube-api-access-pf9lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:20.767710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.767580    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-metrics-certs" pod "656dbefb-91f4-48ce-afe3-742b52686788" (UID: "656dbefb-91f4-48ce-afe3-742b52686788"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:20.767895 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.767874    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-maas-api-tls" pod "656dbefb-91f4-48ce-afe3-742b52686788" (UID: "656dbefb-91f4-48ce-afe3-742b52686788"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:20.885562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.876388    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:20.885562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.876428    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf9lw\" (UniqueName: \"kubernetes.io/projected/656dbefb-91f4-48ce-afe3-742b52686788-kube-api-access-pf9lw\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:20.885562 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:20.876445    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/656dbefb-91f4-48ce-afe3-742b52686788-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:21.499422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.499393    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"
Oct 30 21:54:21.499943 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.499392    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj" event={"ID":"656dbefb-91f4-48ce-afe3-742b52686788","Type":"ContainerDied","Data":"10408b2800b19c97a46e5fdb2ae783e76e49e67dca786842d7b2fd38c5bc4565"}
Oct 30 21:54:21.499943 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.499529    4633 scope.go:122] "RemoveContainer" containerID="81073fc4516b1d92a45f9cf767e99323f46df047cf42f55001df44d8787ce657"
Oct 30 21:54:21.521306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.521277    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl" podUID="5b8f84c7-6449-4add-be75-f2e3819e04a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:21.528281 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.528248    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"]
Oct 30 21:54:21.532630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.532580    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8072a0-756bc8bcd-g5cwj"]
Oct 30 21:54:21.623028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.622992    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:54:21.792384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.792307    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" path="/var/lib/kubelet/pods/30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e/volumes"
Oct 30 21:54:21.792785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:21.792771    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656dbefb-91f4-48ce-afe3-742b52686788" path="/var/lib/kubelet/pods/656dbefb-91f4-48ce-afe3-742b52686788/volumes"
Oct 30 21:54:25.016494 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:25.016450    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:54:28.427846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:28.427799    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"]
Oct 30 21:54:28.428314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:28.428085    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy" containerID="cri-o://7b505ebd678d649e3efb9fde0339d6333775ae755157d6e03080fc9249a9ff84" gracePeriod=30
Oct 30 21:54:31.519645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:31.519603    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl" podUID="5b8f84c7-6449-4add-be75-f2e3819e04a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:33.723378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.723343    4633 generic.go:356] "Generic (PLEG): container finished" podID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerID="7b505ebd678d649e3efb9fde0339d6333775ae755157d6e03080fc9249a9ff84" exitCode=0
Oct 30 21:54:33.723854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.723417    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" event={"ID":"b22001a2-054d-40d7-b536-4f1c26d5dbe2","Type":"ContainerDied","Data":"7b505ebd678d649e3efb9fde0339d6333775ae755157d6e03080fc9249a9ff84"}
Oct 30 21:54:33.723854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.723495    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x" event={"ID":"b22001a2-054d-40d7-b536-4f1c26d5dbe2","Type":"ContainerDied","Data":"07b379c9217ff547819d71709d34fc0d5d1d47e9a3e85e1924b6f5c67208b545"}
Oct 30 21:54:33.723854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.723527    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b379c9217ff547819d71709d34fc0d5d1d47e9a3e85e1924b6f5c67208b545"
Oct 30 21:54:33.723854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.723540    4633 scope.go:122] "RemoveContainer" containerID="eb0f5a34c8e19241f4940518fa62ea118c39e0556508cba1651e7e7033ab937e"
Oct 30 21:54:33.739888 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.739869    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:54:33.742508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.742484    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-kube-api-access-j5fxv\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-kube-api-access-j5fxv\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.742680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.742517    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-credential-socket\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.742680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.742587    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-podinfo\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.742680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.742646    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-socket\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.742880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.742725    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-data\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.742880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.742859    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-credential-socket" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:33.742997 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.742869    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-certs\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.743062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.743026    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-socket" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:33.743062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.743041    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-token\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-token\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.743062 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.743050    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-data" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:33.743227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.743065    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-certs" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:33.743227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.743112    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-envoy\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.743227 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.743166    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istiod-ca-cert\") pod \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\" (UID: \"b22001a2-054d-40d7-b536-4f1c26d5dbe2\") "
Oct 30 21:54:33.743927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.743897    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istiod-ca-cert" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:33.744135 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.744115    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.744219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.744141    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.744219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.744157    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.744219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.744171    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.744219 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.744185    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.745074 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.745049    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-kube-api-access-j5fxv" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "kube-api-access-j5fxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:33.745275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.745256    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-podinfo" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:33.745403 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.745384    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-token" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:33.746335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.746310    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-envoy" pod "b22001a2-054d-40d7-b536-4f1c26d5dbe2" (UID: "b22001a2-054d-40d7-b536-4f1c26d5dbe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:33.872777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.861830    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.872777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.861864    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.872777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.861879    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5fxv\" (UniqueName: \"kubernetes.io/projected/b22001a2-054d-40d7-b536-4f1c26d5dbe2-kube-api-access-j5fxv\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.872777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:33.861894    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22001a2-054d-40d7-b536-4f1c26d5dbe2-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:34.017623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:34.004742    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"]
Oct 30 21:54:34.017623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:34.004972    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy" containerID="cri-o://11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9" gracePeriod=30
Oct 30 21:54:34.732734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:34.732701    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"
Oct 30 21:54:34.771159 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:34.771125    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"]
Oct 30 21:54:34.775079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:34.775054    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8072a0-openshift-default-7d75695589-ksx9x"]
Oct 30 21:54:35.743278 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:35.743240    4633 generic.go:356] "Generic (PLEG): container finished" podID="5b8f84c7-6449-4add-be75-f2e3819e04a7" containerID="a574a24bc46c51fade833df30555b55d2aac37e026ffb0327e0e25e3666d4f46" exitCode=0
Oct 30 21:54:35.743663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:35.743313    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl" event={"ID":"5b8f84c7-6449-4add-be75-f2e3819e04a7","Type":"ContainerDied","Data":"a574a24bc46c51fade833df30555b55d2aac37e026ffb0327e0e25e3666d4f46"}
Oct 30 21:54:35.798102 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:35.798064    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" path="/var/lib/kubelet/pods/b22001a2-054d-40d7-b536-4f1c26d5dbe2/volumes"
Oct 30 21:54:35.894706 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:35.894639    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:54:35.940375 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:35.940330    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8f84c7-6449-4add-be75-f2e3819e04a7-kube-api-access-xjb9h\" (UniqueName: \"kubernetes.io/projected/5b8f84c7-6449-4add-be75-f2e3819e04a7-kube-api-access-xjb9h\") pod \"5b8f84c7-6449-4add-be75-f2e3819e04a7\" (UID: \"5b8f84c7-6449-4add-be75-f2e3819e04a7\") "
Oct 30 21:54:35.940563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:35.940391    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b8f84c7-6449-4add-be75-f2e3819e04a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8f84c7-6449-4add-be75-f2e3819e04a7-plugins-config-volume\") pod \"5b8f84c7-6449-4add-be75-f2e3819e04a7\" (UID: \"5b8f84c7-6449-4add-be75-f2e3819e04a7\") "
Oct 30 21:54:35.941104 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:35.941077    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8f84c7-6449-4add-be75-f2e3819e04a7-plugins-config-volume" pod "5b8f84c7-6449-4add-be75-f2e3819e04a7" (UID: "5b8f84c7-6449-4add-be75-f2e3819e04a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:35.943439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:35.943409    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8f84c7-6449-4add-be75-f2e3819e04a7-kube-api-access-xjb9h" pod "5b8f84c7-6449-4add-be75-f2e3819e04a7" (UID: "5b8f84c7-6449-4add-be75-f2e3819e04a7"). InnerVolumeSpecName "kube-api-access-xjb9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:36.042782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:36.042751    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjb9h\" (UniqueName: \"kubernetes.io/projected/5b8f84c7-6449-4add-be75-f2e3819e04a7-kube-api-access-xjb9h\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:36.042782 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:36.042777    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8f84c7-6449-4add-be75-f2e3819e04a7-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:36.170128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:36.170102    4633 scope.go:122] "RemoveContainer" containerID="7b505ebd678d649e3efb9fde0339d6333775ae755157d6e03080fc9249a9ff84"
Oct 30 21:54:36.181391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:36.181366    4633 scope.go:122] "RemoveContainer" containerID="a574a24bc46c51fade833df30555b55d2aac37e026ffb0327e0e25e3666d4f46"
Oct 30 21:54:36.750205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:36.750168    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl" event={"ID":"5b8f84c7-6449-4add-be75-f2e3819e04a7","Type":"ContainerDied","Data":"8390fcbd572789a210d084d6bbbfb1f2e4fbf910d2007e5520350cb87a91ed11"}
Oct 30 21:54:36.750620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:36.750266    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"
Oct 30 21:54:36.783458 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:36.783426    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"]
Oct 30 21:54:36.787595 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:36.787568    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-87ae68-7b77c74d67-hxzxl"]
Oct 30 21:54:37.792418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:37.792379    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8f84c7-6449-4add-be75-f2e3819e04a7" path="/var/lib/kubelet/pods/5b8f84c7-6449-4add-be75-f2e3819e04a7/volumes"
Oct 30 21:54:39.779392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.779361    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:54:39.787632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.787603    4633 generic.go:356] "Generic (PLEG): container finished" podID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerID="11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9" exitCode=0
Oct 30 21:54:39.787843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.787743    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"
Oct 30 21:54:39.795495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.795429    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" event={"ID":"90cf3712-d04c-4816-9a32-3de435a3f6f7","Type":"ContainerDied","Data":"11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9"}
Oct 30 21:54:39.795495 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.795483    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp" event={"ID":"90cf3712-d04c-4816-9a32-3de435a3f6f7","Type":"ContainerDied","Data":"a53be1cd566c3f8b11da899f9d963ae29eae95bc56bf68d9f46ad2b31c7c06af"}
Oct 30 21:54:39.795682 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.795507    4633 scope.go:122] "RemoveContainer" containerID="11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9"
Oct 30 21:54:39.809296 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.809266    4633 scope.go:122] "RemoveContainer" containerID="11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9"
Oct 30 21:54:39.809646 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:39.809622    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9\": container with ID starting with 11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9 not found: ID does not exist" containerID="11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9"
Oct 30 21:54:39.809823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.809658    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9"} err="failed to get container status \"11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9\": rpc error: code = NotFound desc = could not find container \"11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9\": container with ID starting with 11b1b4ce55cf6b54dd73864321d5a49b9b0b66006b33b67c0b1183a127652af9 not found: ID does not exist"
Oct 30 21:54:39.928767 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.928730    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-podinfo\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.928990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.928834    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-token\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-token\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.928990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.928927    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-socket\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.929140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.929012    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90cf3712-d04c-4816-9a32-3de435a3f6f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf3712-d04c-4816-9a32-3de435a3f6f7-istiod-ca-cert\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.929140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.929073    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-data\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.929140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.929110    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-envoy\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.929322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.929213    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-certs\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.929322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.929268    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-socket" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:39.929442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.929323    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-kube-api-access-bwdcd\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-kube-api-access-bwdcd\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.929442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.929357    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-credential-socket\") pod \"90cf3712-d04c-4816-9a32-3de435a3f6f7\" (UID: \"90cf3712-d04c-4816-9a32-3de435a3f6f7\") "
Oct 30 21:54:39.929847 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.929662    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-data" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:39.930710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.930043    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cf3712-d04c-4816-9a32-3de435a3f6f7-istiod-ca-cert" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:39.930710 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.930437    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-credential-socket" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:39.930890 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.930808    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-certs" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:39.931351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.931330    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:39.931438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.931359    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:39.931438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.931361    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-podinfo" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:39.931438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.931375    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90cf3712-d04c-4816-9a32-3de435a3f6f7-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:39.931438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.931390    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:39.931438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.931404    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:39.932088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.932026    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-token" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:39.933762 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.933738    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-kube-api-access-bwdcd" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "kube-api-access-bwdcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:39.934920 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:39.934903    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-envoy" pod "90cf3712-d04c-4816-9a32-3de435a3f6f7" (UID: "90cf3712-d04c-4816-9a32-3de435a3f6f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.032819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:40.032786    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwdcd\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-kube-api-access-bwdcd\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.032819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:40.032818    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.032819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:40.032828    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.033094 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:40.032838    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90cf3712-d04c-4816-9a32-3de435a3f6f7-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.114418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:40.114384    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"]
Oct 30 21:54:40.121632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:40.121607    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-4lkpp"]
Oct 30 21:54:41.707519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:41.707454    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"]
Oct 30 21:54:41.708504 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:41.708450    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy" containerID="cri-o://a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca" gracePeriod=30
Oct 30 21:54:41.794801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:41.794768    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" path="/var/lib/kubelet/pods/90cf3712-d04c-4816-9a32-3de435a3f6f7/volumes"
Oct 30 21:54:46.601201 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:46.601105    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"]
Oct 30 21:54:46.601650 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:46.601452    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z" podUID="475d261a-7603-40c1-8e1a-6e62ee35a5b9" containerName="maas-api" containerID="cri-o://5d364f18bd85c35e323cc94959a03af36ee571e8cb00b666ccc18b46e918ff54" gracePeriod=30
Oct 30 21:54:47.050213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.050171    4633 generic.go:356] "Generic (PLEG): container finished" podID="475d261a-7603-40c1-8e1a-6e62ee35a5b9" containerID="5d364f18bd85c35e323cc94959a03af36ee571e8cb00b666ccc18b46e918ff54" exitCode=0
Oct 30 21:54:47.050422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.050287    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z" event={"ID":"475d261a-7603-40c1-8e1a-6e62ee35a5b9","Type":"ContainerDied","Data":"5d364f18bd85c35e323cc94959a03af36ee571e8cb00b666ccc18b46e918ff54"}
Oct 30 21:54:47.153214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.152823    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:54:47.280423 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.280388    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:54:47.350343 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.350254    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-metrics-certs\") pod \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") "
Oct 30 21:54:47.350535 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.350459    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475d261a-7603-40c1-8e1a-6e62ee35a5b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/475d261a-7603-40c1-8e1a-6e62ee35a5b9-openshift-service-ca\") pod \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") "
Oct 30 21:54:47.350628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.350611    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475d261a-7603-40c1-8e1a-6e62ee35a5b9-kube-api-access-krh2n\" (UniqueName: \"kubernetes.io/projected/475d261a-7603-40c1-8e1a-6e62ee35a5b9-kube-api-access-krh2n\") pod \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") "
Oct 30 21:54:47.350718 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.350683    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-maas-api-tls\") pod \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\" (UID: \"475d261a-7603-40c1-8e1a-6e62ee35a5b9\") "
Oct 30 21:54:47.353911 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.353873    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475d261a-7603-40c1-8e1a-6e62ee35a5b9-openshift-service-ca" pod "475d261a-7603-40c1-8e1a-6e62ee35a5b9" (UID: "475d261a-7603-40c1-8e1a-6e62ee35a5b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:47.356132 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.356091    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-maas-api-tls" pod "475d261a-7603-40c1-8e1a-6e62ee35a5b9" (UID: "475d261a-7603-40c1-8e1a-6e62ee35a5b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:47.356132 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.356099    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475d261a-7603-40c1-8e1a-6e62ee35a5b9-kube-api-access-krh2n" pod "475d261a-7603-40c1-8e1a-6e62ee35a5b9" (UID: "475d261a-7603-40c1-8e1a-6e62ee35a5b9"). InnerVolumeSpecName "kube-api-access-krh2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:47.357783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.357752    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-metrics-certs" pod "475d261a-7603-40c1-8e1a-6e62ee35a5b9" (UID: "475d261a-7603-40c1-8e1a-6e62ee35a5b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:47.452608 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.452571    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-envoy\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.452850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.452740    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-istio-token\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-istio-token\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.452850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.452813    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-data\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.452973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.452920    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de810876-cea8-4249-b707-7d20744b27ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de810876-cea8-4249-b707-7d20744b27ad-istiod-ca-cert\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.452973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.452967    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-socket\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.453090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453060    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-certs\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.453090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453060    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-data" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.453205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453127    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-kube-api-access-kwpcr\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-kube-api-access-kwpcr\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.453205 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453184    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de810876-cea8-4249-b707-7d20744b27ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de810876-cea8-4249-b707-7d20744b27ad-istio-podinfo\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.453330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453218    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-credential-socket\") pod \"de810876-cea8-4249-b707-7d20744b27ad\" (UID: \"de810876-cea8-4249-b707-7d20744b27ad\") "
Oct 30 21:54:47.453330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453287    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-socket" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.453330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453310    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-certs" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.453545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453365    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de810876-cea8-4249-b707-7d20744b27ad-istiod-ca-cert" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:47.453754 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.453635    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-credential-socket" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.454087 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454063    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krh2n\" (UniqueName: \"kubernetes.io/projected/475d261a-7603-40c1-8e1a-6e62ee35a5b9-kube-api-access-krh2n\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.454187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454092    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.454187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454108    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.454187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454126    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/475d261a-7603-40c1-8e1a-6e62ee35a5b9-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.454187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454140    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.454187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454155    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/475d261a-7603-40c1-8e1a-6e62ee35a5b9-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.454187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454173    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de810876-cea8-4249-b707-7d20744b27ad-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.454513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454190    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.454513 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.454208    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.455265 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.455240    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-envoy" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.455351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.455304    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-istio-token" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:47.455393 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.455386    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de810876-cea8-4249-b707-7d20744b27ad-istio-podinfo" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:47.455507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.455491    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-kube-api-access-kwpcr" pod "de810876-cea8-4249-b707-7d20744b27ad" (UID: "de810876-cea8-4249-b707-7d20744b27ad"). InnerVolumeSpecName "kube-api-access-kwpcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:47.555946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.555908    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwpcr\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-kube-api-access-kwpcr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.555946 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.555945    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de810876-cea8-4249-b707-7d20744b27ad-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.556179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.555959    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de810876-cea8-4249-b707-7d20744b27ad-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.556179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:47.555973    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de810876-cea8-4249-b707-7d20744b27ad-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:48.062140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.062092    4633 generic.go:356] "Generic (PLEG): container finished" podID="de810876-cea8-4249-b707-7d20744b27ad" containerID="a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca" exitCode=0
Oct 30 21:54:48.062654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.062166    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"
Oct 30 21:54:48.062654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.062166    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" event={"ID":"de810876-cea8-4249-b707-7d20744b27ad","Type":"ContainerDied","Data":"a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca"}
Oct 30 21:54:48.062654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.062265    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc" event={"ID":"de810876-cea8-4249-b707-7d20744b27ad","Type":"ContainerDied","Data":"34ff15e92621493bfa27e5efefeef2f5753e7aa522a89291849b16b37993349d"}
Oct 30 21:54:48.062654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.062411    4633 scope.go:122] "RemoveContainer" containerID="a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca"
Oct 30 21:54:48.065651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.065626    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z" event={"ID":"475d261a-7603-40c1-8e1a-6e62ee35a5b9","Type":"ContainerDied","Data":"9daa6e6a5e8a4299058901b82e2f46f542668fafd68fab877a06fd783ab963f0"}
Oct 30 21:54:48.065651 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.065644    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"
Oct 30 21:54:48.084830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.084807    4633 scope.go:122] "RemoveContainer" containerID="a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca"
Oct 30 21:54:48.086119 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:48.086085    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca\": container with ID starting with a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca not found: ID does not exist" containerID="a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca"
Oct 30 21:54:48.086235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.086148    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca"} err="failed to get container status \"a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca\": rpc error: code = NotFound desc = could not find container \"a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca\": container with ID starting with a1f201de554e808ba254120698972e08cffbd5e18b76e838f26b97ce360e00ca not found: ID does not exist"
Oct 30 21:54:48.086235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.086172    4633 scope.go:122] "RemoveContainer" containerID="5d364f18bd85c35e323cc94959a03af36ee571e8cb00b666ccc18b46e918ff54"
Oct 30 21:54:48.101091 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.101030    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"]
Oct 30 21:54:48.114435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.114397    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-lndlc"]
Oct 30 21:54:48.137314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.137281    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"]
Oct 30 21:54:48.151234 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:48.151202    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-36a099-5fd5d68bc9-cx48z"]
Oct 30 21:54:49.792923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:49.792877    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475d261a-7603-40c1-8e1a-6e62ee35a5b9" path="/var/lib/kubelet/pods/475d261a-7603-40c1-8e1a-6e62ee35a5b9/volumes"
Oct 30 21:54:49.793489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:49.793472    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de810876-cea8-4249-b707-7d20744b27ad" path="/var/lib/kubelet/pods/de810876-cea8-4249-b707-7d20744b27ad/volumes"
Oct 30 21:54:50.559580 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:50.559545    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"]
Oct 30 21:54:50.559840 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:50.559813    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy" containerID="cri-o://1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b" gracePeriod=30
Oct 30 21:54:52.643110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:52.643076    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"]
Oct 30 21:54:53.402480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:53.402434    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"]
Oct 30 21:54:53.403119 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:53.403061    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy" containerID="cri-o://352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a" gracePeriod=30
Oct 30 21:54:55.015691 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:55.015638    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:54:55.735123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:55.734461    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"]
Oct 30 21:54:55.735123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:55.734745    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" containerID="cri-o://a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb" gracePeriod=30
Oct 30 21:54:55.748456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:55.746735    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:55.748456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:55.746791    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:56.053994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.053959    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:54:56.090316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.090265    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-token\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-token\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.090503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.090404    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-data\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.090565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.090499    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-credential-socket\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.090620 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.090575    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df40741b-59ed-4b6d-95df-8b515a02eb61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40741b-59ed-4b6d-95df-8b515a02eb61-istiod-ca-cert\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.090705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.090649    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-kube-api-access-qffw8\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-kube-api-access-qffw8\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.090815 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.090799    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-envoy\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.090948 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.090933    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-socket\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.091007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.090966    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-podinfo\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.091059 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.091029    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-certs\") pod \"df40741b-59ed-4b6d-95df-8b515a02eb61\" (UID: \"df40741b-59ed-4b6d-95df-8b515a02eb61\") "
Oct 30 21:54:56.091148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.091119    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-credential-socket" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:56.091360 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.091338    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-data" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:56.091998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.091925    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.092131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.092007    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.092131 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.092015    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-socket" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:56.092368 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.092332    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df40741b-59ed-4b6d-95df-8b515a02eb61-istiod-ca-cert" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:56.092637 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.092612    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-certs" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:56.094981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.094949    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-envoy" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:56.095730 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.095696    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-kube-api-access-qffw8" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "kube-api-access-qffw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:56.096531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.096502    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-podinfo" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:56.096761 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.096699    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-token" pod "df40741b-59ed-4b6d-95df-8b515a02eb61" (UID: "df40741b-59ed-4b6d-95df-8b515a02eb61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:56.156370 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.156337    4633 generic.go:356] "Generic (PLEG): container finished" podID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerID="1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b" exitCode=0
Oct 30 21:54:56.156551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.156387    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"
Oct 30 21:54:56.156551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.156421    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" event={"ID":"df40741b-59ed-4b6d-95df-8b515a02eb61","Type":"ContainerDied","Data":"1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b"}
Oct 30 21:54:56.156551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.156476    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c" event={"ID":"df40741b-59ed-4b6d-95df-8b515a02eb61","Type":"ContainerDied","Data":"8724e3fa659baf1dcecc9b03ccde42f5c24852aa67d03eede388d673ee8babfd"}
Oct 30 21:54:56.156551 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.156498    4633 scope.go:122] "RemoveContainer" containerID="1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b"
Oct 30 21:54:56.176565 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.176543    4633 scope.go:122] "RemoveContainer" containerID="1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b"
Oct 30 21:54:56.176970 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:56.176949    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b\": container with ID starting with 1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b not found: ID does not exist" containerID="1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b"
Oct 30 21:54:56.177058 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.176982    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b"} err="failed to get container status \"1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b\": rpc error: code = NotFound desc = could not find container \"1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b\": container with ID starting with 1bd2b04dd706c0b288fb947f68cd48bee9be24bfd31521c01f2b086eababed4b not found: ID does not exist"
Oct 30 21:54:56.192659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.192631    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"]
Oct 30 21:54:56.193297 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.193277    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.193348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.193301    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.193348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.193312    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.193348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.193322    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.193348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.193331    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40741b-59ed-4b6d-95df-8b515a02eb61-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.193348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.193340    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qffw8\" (UniqueName: \"kubernetes.io/projected/df40741b-59ed-4b6d-95df-8b515a02eb61-kube-api-access-qffw8\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.193525 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.193350    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40741b-59ed-4b6d-95df-8b515a02eb61-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.200864 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:56.200838    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-6pk8c"]
Oct 30 21:54:57.644123 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:57.644075    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl" podUID="d4fe890f-059c-478e-a61a-8f8dc565b237" containerName="payload-processing" containerID="cri-o://edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c" gracePeriod=25
Oct 30 21:54:57.793366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:57.793325    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" path="/var/lib/kubelet/pods/df40741b-59ed-4b6d-95df-8b515a02eb61/volumes"
Oct 30 21:54:57.964335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:57.964309    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:54:58.032818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.032259    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fe890f-059c-478e-a61a-8f8dc565b237-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fe890f-059c-478e-a61a-8f8dc565b237-plugins-config-volume\") pod \"d4fe890f-059c-478e-a61a-8f8dc565b237\" (UID: \"d4fe890f-059c-478e-a61a-8f8dc565b237\") "
Oct 30 21:54:58.032818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.032340    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fe890f-059c-478e-a61a-8f8dc565b237-kube-api-access-bvp5g\" (UniqueName: \"kubernetes.io/projected/d4fe890f-059c-478e-a61a-8f8dc565b237-kube-api-access-bvp5g\") pod \"d4fe890f-059c-478e-a61a-8f8dc565b237\" (UID: \"d4fe890f-059c-478e-a61a-8f8dc565b237\") "
Oct 30 21:54:58.036856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.033843    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fe890f-059c-478e-a61a-8f8dc565b237-plugins-config-volume" pod "d4fe890f-059c-478e-a61a-8f8dc565b237" (UID: "d4fe890f-059c-478e-a61a-8f8dc565b237"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:58.049632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.048879    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fe890f-059c-478e-a61a-8f8dc565b237-kube-api-access-bvp5g" pod "d4fe890f-059c-478e-a61a-8f8dc565b237" (UID: "d4fe890f-059c-478e-a61a-8f8dc565b237"). InnerVolumeSpecName "kube-api-access-bvp5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:58.134391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.134359    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fe890f-059c-478e-a61a-8f8dc565b237-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.134391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.134388    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvp5g\" (UniqueName: \"kubernetes.io/projected/d4fe890f-059c-478e-a61a-8f8dc565b237-kube-api-access-bvp5g\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.178057 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.178020    4633 generic.go:356] "Generic (PLEG): container finished" podID="d4fe890f-059c-478e-a61a-8f8dc565b237" containerID="edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c" exitCode=0
Oct 30 21:54:58.178228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.178081    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"
Oct 30 21:54:58.178228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.178104    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl" event={"ID":"d4fe890f-059c-478e-a61a-8f8dc565b237","Type":"ContainerDied","Data":"edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c"}
Oct 30 21:54:58.178228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.178166    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl" event={"ID":"d4fe890f-059c-478e-a61a-8f8dc565b237","Type":"ContainerDied","Data":"d167e3083cb1f33697378433b1612f1eef8faa7458621b0824595173b3d0b8ac"}
Oct 30 21:54:58.178228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.178188    4633 scope.go:122] "RemoveContainer" containerID="edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c"
Oct 30 21:54:58.189885 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.189863    4633 scope.go:122] "RemoveContainer" containerID="edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c"
Oct 30 21:54:58.190208 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:58.190186    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c\": container with ID starting with edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c not found: ID does not exist" containerID="edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c"
Oct 30 21:54:58.190308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.190221    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c"} err="failed to get container status \"edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c\": rpc error: code = NotFound desc = could not find container \"edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c\": container with ID starting with edf1c520953a0ab1769879ceece076236e54632458dc999d50322a875082d85c not found: ID does not exist"
Oct 30 21:54:58.227257 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.227223    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"]
Oct 30 21:54:58.237107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.237075    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15e79c-566988c7c9-sv4bl"]
Oct 30 21:54:58.688168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.688143    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:54:58.742392 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742307    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-data\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.742559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742461    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-credential-socket\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.742559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742505    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-podinfo\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.742559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742538    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istiod-ca-cert\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.742814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742599    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-certs\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.742814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742628    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-data" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:58.742814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742656    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-envoy\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.742814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742799    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-socket\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.743073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742827    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-token\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-token\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.743073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742809    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-credential-socket" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:58.743073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.742924    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-kube-api-access-vv5v2\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-kube-api-access-vv5v2\") pod \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\" (UID: \"50cb03b6-5f8c-4e62-9bb8-4285acb19238\") "
Oct 30 21:54:58.743073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.743020    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-socket" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:58.743309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.743118    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-certs" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:58.743309 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.743269    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istiod-ca-cert" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:58.743534 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.743517    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.743827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.743539    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.743827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.743554    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.743827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.743566    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.743827 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.743577    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.745547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.745522    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-kube-api-access-vv5v2" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "kube-api-access-vv5v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:58.747064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.747020    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-podinfo" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:58.747064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.747032    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-envoy" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:58.748363 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.748341    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-token" pod "50cb03b6-5f8c-4e62-9bb8-4285acb19238" (UID: "50cb03b6-5f8c-4e62-9bb8-4285acb19238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:58.846924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.846889    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.846924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.846921    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.846924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.846934    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.846924 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:58.846948    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv5v2\" (UniqueName: \"kubernetes.io/projected/50cb03b6-5f8c-4e62-9bb8-4285acb19238-kube-api-access-vv5v2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:59.213987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.213826    4633 generic.go:356] "Generic (PLEG): container finished" podID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerID="352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a" exitCode=0
Oct 30 21:54:59.214200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.214007    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" event={"ID":"50cb03b6-5f8c-4e62-9bb8-4285acb19238","Type":"ContainerDied","Data":"352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a"}
Oct 30 21:54:59.214200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.214061    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx" event={"ID":"50cb03b6-5f8c-4e62-9bb8-4285acb19238","Type":"ContainerDied","Data":"4ceb37acfa39ff721baa1fa31d31bdb055d0d2edd7b3ba26e82f99a93814ccd1"}
Oct 30 21:54:59.214200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.214088    4633 scope.go:122] "RemoveContainer" containerID="352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a"
Oct 30 21:54:59.214414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.214208    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"
Oct 30 21:54:59.228073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.228050    4633 scope.go:122] "RemoveContainer" containerID="352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a"
Oct 30 21:54:59.228434 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:54:59.228415    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a\": container with ID starting with 352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a not found: ID does not exist" containerID="352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a"
Oct 30 21:54:59.228487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.228444    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a"} err="failed to get container status \"352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a\": rpc error: code = NotFound desc = could not find container \"352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a\": container with ID starting with 352635735e6b13bd1025226186f7946a2f6acd5d0ba5ab0184270d13570ebc5a not found: ID does not exist"
Oct 30 21:54:59.246213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.246177    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"]
Oct 30 21:54:59.250448 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.250419    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-npwnx"]
Oct 30 21:54:59.792696 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.792641    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" path="/var/lib/kubelet/pods/50cb03b6-5f8c-4e62-9bb8-4285acb19238/volumes"
Oct 30 21:54:59.793168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:54:59.793151    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fe890f-059c-478e-a61a-8f8dc565b237" path="/var/lib/kubelet/pods/d4fe890f-059c-478e-a61a-8f8dc565b237/volumes"
Oct 30 21:55:01.012097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.012067    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:55:01.078235 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078153    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-envoy\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078267    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-socket\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078316    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-data\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078536 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078413    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-certs\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078536 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078506    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-kube-api-access-pz9hb\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-kube-api-access-pz9hb\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078539    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e5d59e-3940-42c0-906f-38bad87cb8ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e5d59e-3940-42c0-906f-38bad87cb8ff-istiod-ca-cert\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078640    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-token\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-token\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078795 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078711    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-podinfo\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078806    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-credential-socket\") pod \"07e5d59e-3940-42c0-906f-38bad87cb8ff\" (UID: \"07e5d59e-3940-42c0-906f-38bad87cb8ff\") "
Oct 30 21:55:01.078918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.078865    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-data" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.079116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.079091    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-socket" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.079431 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.079412    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.079538 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.079435    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.079720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.079652    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-certs" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.080150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.079947    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-credential-socket" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.080150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.080117    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e5d59e-3940-42c0-906f-38bad87cb8ff-istiod-ca-cert" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:01.082530 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.082483    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-kube-api-access-pz9hb" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "kube-api-access-pz9hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:01.083217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.083158    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-podinfo" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:01.084545 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.084525    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-envoy" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.084991 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.084963    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-token" pod "07e5d59e-3940-42c0-906f-38bad87cb8ff" (UID: "07e5d59e-3940-42c0-906f-38bad87cb8ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:01.181255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.181211    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.181255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.181247    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9hb\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-kube-api-access-pz9hb\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.181255 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.181259    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07e5d59e-3940-42c0-906f-38bad87cb8ff-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.181509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.181269    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.181509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.181278    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.181509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.181287    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.181509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.181296    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07e5d59e-3940-42c0-906f-38bad87cb8ff-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.236037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.236005    4633 generic.go:356] "Generic (PLEG): container finished" podID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerID="a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb" exitCode=0
Oct 30 21:55:01.236241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.236058    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" event={"ID":"07e5d59e-3940-42c0-906f-38bad87cb8ff","Type":"ContainerDied","Data":"a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb"}
Oct 30 21:55:01.236241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.236073    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"
Oct 30 21:55:01.236241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.236105    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7" event={"ID":"07e5d59e-3940-42c0-906f-38bad87cb8ff","Type":"ContainerDied","Data":"7027b821056b55e11f44be8cc6eaa47032d0de405acbc100381fc761ca376419"}
Oct 30 21:55:01.236241 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.236129    4633 scope.go:122] "RemoveContainer" containerID="a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb"
Oct 30 21:55:01.248746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.248720    4633 scope.go:122] "RemoveContainer" containerID="a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb"
Oct 30 21:55:01.249120 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:01.249099    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb\": container with ID starting with a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb not found: ID does not exist" containerID="a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb"
Oct 30 21:55:01.249187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.249128    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb"} err="failed to get container status \"a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb\": rpc error: code = NotFound desc = could not find container \"a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb\": container with ID starting with a00fbbc03e031caa2222d2272e28eed64dbb3d54828cc47719abeb8ef51004bb not found: ID does not exist"
Oct 30 21:55:01.265220 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.265186    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"]
Oct 30 21:55:01.269077 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.269047    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-vwjc7"]
Oct 30 21:55:01.346184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.346083    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"]
Oct 30 21:55:01.803643 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:01.803602    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" path="/var/lib/kubelet/pods/07e5d59e-3940-42c0-906f-38bad87cb8ff/volumes"
Oct 30 21:55:06.347522 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:06.347480    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7" podUID="95f4f687-e025-46fd-85a5-9a93cfef9cc6" containerName="payload-processing" containerID="cri-o://d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43" gracePeriod=25
Oct 30 21:55:06.650048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:06.650007    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:55:06.795157 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:06.795121    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f4f687-e025-46fd-85a5-9a93cfef9cc6-kube-api-access-ppzcm\" (UniqueName: \"kubernetes.io/projected/95f4f687-e025-46fd-85a5-9a93cfef9cc6-kube-api-access-ppzcm\") pod \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\" (UID: \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\") "
Oct 30 21:55:06.795394 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:06.795378    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f4f687-e025-46fd-85a5-9a93cfef9cc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f4f687-e025-46fd-85a5-9a93cfef9cc6-plugins-config-volume\") pod \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\" (UID: \"95f4f687-e025-46fd-85a5-9a93cfef9cc6\") "
Oct 30 21:55:06.795874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:06.795848    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f4f687-e025-46fd-85a5-9a93cfef9cc6-plugins-config-volume" pod "95f4f687-e025-46fd-85a5-9a93cfef9cc6" (UID: "95f4f687-e025-46fd-85a5-9a93cfef9cc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:06.797498 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:06.797473    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f4f687-e025-46fd-85a5-9a93cfef9cc6-kube-api-access-ppzcm" pod "95f4f687-e025-46fd-85a5-9a93cfef9cc6" (UID: "95f4f687-e025-46fd-85a5-9a93cfef9cc6"). InnerVolumeSpecName "kube-api-access-ppzcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:06.897327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:06.897247    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95f4f687-e025-46fd-85a5-9a93cfef9cc6-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:06.897327 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:06.897281    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppzcm\" (UniqueName: \"kubernetes.io/projected/95f4f687-e025-46fd-85a5-9a93cfef9cc6-kube-api-access-ppzcm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:07.285181 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.285142    4633 generic.go:356] "Generic (PLEG): container finished" podID="95f4f687-e025-46fd-85a5-9a93cfef9cc6" containerID="d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43" exitCode=0
Oct 30 21:55:07.285387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.285189    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"
Oct 30 21:55:07.285387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.285218    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7" event={"ID":"95f4f687-e025-46fd-85a5-9a93cfef9cc6","Type":"ContainerDied","Data":"d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43"}
Oct 30 21:55:07.285387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.285273    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7" event={"ID":"95f4f687-e025-46fd-85a5-9a93cfef9cc6","Type":"ContainerDied","Data":"a69b8c31933561a9b9096304d4ed1f92ee4078515315e06901d751e41bb878ab"}
Oct 30 21:55:07.285387 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.285294    4633 scope.go:122] "RemoveContainer" containerID="d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43"
Oct 30 21:55:07.296998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.296980    4633 scope.go:122] "RemoveContainer" containerID="d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43"
Oct 30 21:55:07.297315 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:07.297295    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43\": container with ID starting with d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43 not found: ID does not exist" containerID="d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43"
Oct 30 21:55:07.297362 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.297328    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43"} err="failed to get container status \"d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43\": rpc error: code = NotFound desc = could not find container \"d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43\": container with ID starting with d99205e8b9da523cd73aaa68f1d94b9a62f5d1174cef55be63fcb4c8e3990b43 not found: ID does not exist"
Oct 30 21:55:07.345315 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.345273    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"]
Oct 30 21:55:07.350213 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.350184    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f9e5a7-845fcd9c47-74jv7"]
Oct 30 21:55:07.792329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.792298    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f4f687-e025-46fd-85a5-9a93cfef9cc6" path="/var/lib/kubelet/pods/95f4f687-e025-46fd-85a5-9a93cfef9cc6/volumes"
Oct 30 21:55:07.868607 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:07.868575    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"]
Oct 30 21:55:10.365391 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:10.365356    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:55:14.995737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:14.995700    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"]
Oct 30 21:55:14.996237 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:14.995945    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy" containerID="cri-o://7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33" gracePeriod=30
Oct 30 21:55:18.297649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:18.297607    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"]
Oct 30 21:55:18.298154 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:18.297842    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" containerID="cri-o://c6ed724bd4f6ea3abec644aa8056d034487cc465b01f3c010d9e43e502bb8544" gracePeriod=30
Oct 30 21:55:20.159075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.159040    4633 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 30 21:55:20.159479 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.159127    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 30 21:55:20.308254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.308223    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:55:20.409269 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409233    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istiod-ca-cert\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409361    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-kube-api-access-p9nq5\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-kube-api-access-p9nq5\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409390    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-data\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409427    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-certs\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409464 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409448    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-credential-socket\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409467    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-envoy\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409514    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-podinfo\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409554    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-socket\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409649    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-token\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-token\") pod \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\" (UID: \"3153c1af-b6f5-482d-bcc6-c577c5ad697d\") "
Oct 30 21:55:20.409992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409785    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-data" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.409992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409769    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-certs" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.409992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409807    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-credential-socket" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.409992 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.409833    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istiod-ca-cert" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:20.410204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.410028    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-socket" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.410359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.410339    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.410432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.410363    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.410432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.410378    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.410432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.410392    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.410432 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.410406    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.412187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.412153    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-podinfo" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:20.412303 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.412193    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-envoy" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.412568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.412547    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-token" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:20.412632 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.412578    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-kube-api-access-p9nq5" pod "3153c1af-b6f5-482d-bcc6-c577c5ad697d" (UID: "3153c1af-b6f5-482d-bcc6-c577c5ad697d"). InnerVolumeSpecName "kube-api-access-p9nq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:20.482170 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.482139    4633 generic.go:356] "Generic (PLEG): container finished" podID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerID="7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33" exitCode=0
Oct 30 21:55:20.482353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.482204    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"
Oct 30 21:55:20.482353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.482246    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" event={"ID":"3153c1af-b6f5-482d-bcc6-c577c5ad697d","Type":"ContainerDied","Data":"7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33"}
Oct 30 21:55:20.482353 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.482316    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p" event={"ID":"3153c1af-b6f5-482d-bcc6-c577c5ad697d","Type":"ContainerDied","Data":"015d5c55010d2fa49afe9304c1c972dee186836eecf142dc0cee68420b9a6b21"}
Oct 30 21:55:20.482484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.482355    4633 scope.go:122] "RemoveContainer" containerID="7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33"
Oct 30 21:55:20.495206 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.494861    4633 scope.go:122] "RemoveContainer" containerID="bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b"
Oct 30 21:55:20.504902 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.504868    4633 scope.go:122] "RemoveContainer" containerID="7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33"
Oct 30 21:55:20.505171 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:20.505147    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33\": container with ID starting with 7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33 not found: ID does not exist" containerID="7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33"
Oct 30 21:55:20.505229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.505182    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33"} err="failed to get container status \"7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33\": rpc error: code = NotFound desc = could not find container \"7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33\": container with ID starting with 7a23c58117e7850654a14385e6cbacef1a4083c33c62e01d7aff221e547c2e33 not found: ID does not exist"
Oct 30 21:55:20.505229 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.505200    4633 scope.go:122] "RemoveContainer" containerID="bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b"
Oct 30 21:55:20.505462 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:20.505445    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b\": container with ID starting with bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b not found: ID does not exist" containerID="bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b"
Oct 30 21:55:20.505514 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.505465    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b"} err="failed to get container status \"bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b\": rpc error: code = NotFound desc = could not find container \"bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b\": container with ID starting with bab1edab3b64a0f7c19cfb81b015d0df111e18d7a3cbd9a3a4dbb45230b5644b not found: ID does not exist"
Oct 30 21:55:20.512161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.512137    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.512161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.512163    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9nq5\" (UniqueName: \"kubernetes.io/projected/3153c1af-b6f5-482d-bcc6-c577c5ad697d-kube-api-access-p9nq5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.512323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.512174    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.512323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.512184    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3153c1af-b6f5-482d-bcc6-c577c5ad697d-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.513618 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.513596    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"]
Oct 30 21:55:20.519751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.519722    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-qcp7p"]
Oct 30 21:55:20.713263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.713234    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:55:20.713438 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:20.713296    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:55:21.833802 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:21.833766    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" path="/var/lib/kubelet/pods/3153c1af-b6f5-482d-bcc6-c577c5ad697d/volumes"
Oct 30 21:55:22.567988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:22.567957    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"]
Oct 30 21:55:22.568284 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:22.568229    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72" podUID="d5d863f5-b05e-4922-9f70-015323992554" containerName="maas-api" containerID="cri-o://1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d" gracePeriod=30
Oct 30 21:55:22.934701 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:22.931767    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2" podUID="9645737c-61cc-475d-9832-d5dda1b7dee9" containerName="main" containerID="cri-o://fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948" gracePeriod=45
Oct 30 21:55:22.972616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:22.972574    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2" podUID="9645737c-61cc-475d-9832-d5dda1b7dee9" containerName="main" probeResult="failure" output="Get \"https://10.133.1.19:8000/ready\": dial tcp 10.133.1.19:8000: connect: connection refused"
Oct 30 21:55:23.072916 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.070357    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:55:23.177373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.177326    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-metrics-certs\") pod \"d5d863f5-b05e-4922-9f70-015323992554\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") "
Oct 30 21:55:23.177564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.177501    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d863f5-b05e-4922-9f70-015323992554-kube-api-access-jwht5\" (UniqueName: \"kubernetes.io/projected/d5d863f5-b05e-4922-9f70-015323992554-kube-api-access-jwht5\") pod \"d5d863f5-b05e-4922-9f70-015323992554\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") "
Oct 30 21:55:23.177644 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.177608    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d863f5-b05e-4922-9f70-015323992554-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d863f5-b05e-4922-9f70-015323992554-openshift-service-ca\") pod \"d5d863f5-b05e-4922-9f70-015323992554\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") "
Oct 30 21:55:23.177741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.177643    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-maas-api-tls\") pod \"d5d863f5-b05e-4922-9f70-015323992554\" (UID: \"d5d863f5-b05e-4922-9f70-015323992554\") "
Oct 30 21:55:23.179071 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.179045    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d863f5-b05e-4922-9f70-015323992554-openshift-service-ca" pod "d5d863f5-b05e-4922-9f70-015323992554" (UID: "d5d863f5-b05e-4922-9f70-015323992554"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:23.182092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.181775    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d863f5-b05e-4922-9f70-015323992554-kube-api-access-jwht5" pod "d5d863f5-b05e-4922-9f70-015323992554" (UID: "d5d863f5-b05e-4922-9f70-015323992554"). InnerVolumeSpecName "kube-api-access-jwht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:23.182092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.181897    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-metrics-certs" pod "d5d863f5-b05e-4922-9f70-015323992554" (UID: "d5d863f5-b05e-4922-9f70-015323992554"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:23.183196 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.183169    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-maas-api-tls" pod "d5d863f5-b05e-4922-9f70-015323992554" (UID: "d5d863f5-b05e-4922-9f70-015323992554"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:23.279454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.279425    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5d863f5-b05e-4922-9f70-015323992554-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.279454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.279451    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.279454 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.279460    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5d863f5-b05e-4922-9f70-015323992554-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.279788 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.279469    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwht5\" (UniqueName: \"kubernetes.io/projected/d5d863f5-b05e-4922-9f70-015323992554-kube-api-access-jwht5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.287277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.287253    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:55:23.380926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.380896    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-model-cache\") pod \"9645737c-61cc-475d-9832-d5dda1b7dee9\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") "
Oct 30 21:55:23.381109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.380969    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9645737c-61cc-475d-9832-d5dda1b7dee9-kube-api-access-9758w\" (UniqueName: \"kubernetes.io/projected/9645737c-61cc-475d-9832-d5dda1b7dee9-kube-api-access-9758w\") pod \"9645737c-61cc-475d-9832-d5dda1b7dee9\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") "
Oct 30 21:55:23.381109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.380997    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-dshm\") pod \"9645737c-61cc-475d-9832-d5dda1b7dee9\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") "
Oct 30 21:55:23.381109 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381034    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-home\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-home\") pod \"9645737c-61cc-475d-9832-d5dda1b7dee9\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") "
Oct 30 21:55:23.381286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381143    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-tmp-dir\") pod \"9645737c-61cc-475d-9832-d5dda1b7dee9\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") "
Oct 30 21:55:23.381286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381258    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9645737c-61cc-475d-9832-d5dda1b7dee9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9645737c-61cc-475d-9832-d5dda1b7dee9-tls-certs\") pod \"9645737c-61cc-475d-9832-d5dda1b7dee9\" (UID: \"9645737c-61cc-475d-9832-d5dda1b7dee9\") "
Oct 30 21:55:23.381286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381268    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-model-cache" pod "9645737c-61cc-475d-9832-d5dda1b7dee9" (UID: "9645737c-61cc-475d-9832-d5dda1b7dee9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.381527 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381503    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-home" pod "9645737c-61cc-475d-9832-d5dda1b7dee9" (UID: "9645737c-61cc-475d-9832-d5dda1b7dee9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.381708 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381659    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-tmp-dir" pod "9645737c-61cc-475d-9832-d5dda1b7dee9" (UID: "9645737c-61cc-475d-9832-d5dda1b7dee9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.381915 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381890    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.382005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381923    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.382005 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.381961    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.384063 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.384032    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9645737c-61cc-475d-9832-d5dda1b7dee9-tls-certs" pod "9645737c-61cc-475d-9832-d5dda1b7dee9" (UID: "9645737c-61cc-475d-9832-d5dda1b7dee9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:23.384180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.384156    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-dshm" pod "9645737c-61cc-475d-9832-d5dda1b7dee9" (UID: "9645737c-61cc-475d-9832-d5dda1b7dee9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.384321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.384299    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9645737c-61cc-475d-9832-d5dda1b7dee9-kube-api-access-9758w" pod "9645737c-61cc-475d-9832-d5dda1b7dee9" (UID: "9645737c-61cc-475d-9832-d5dda1b7dee9"). InnerVolumeSpecName "kube-api-access-9758w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:23.484356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.484256    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9645737c-61cc-475d-9832-d5dda1b7dee9-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.484356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.484299    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9758w\" (UniqueName: \"kubernetes.io/projected/9645737c-61cc-475d-9832-d5dda1b7dee9-kube-api-access-9758w\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.484356 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.484319    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9645737c-61cc-475d-9832-d5dda1b7dee9-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.512557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.512509    4633 generic.go:356] "Generic (PLEG): container finished" podID="d5d863f5-b05e-4922-9f70-015323992554" containerID="1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d" exitCode=0
Oct 30 21:55:23.512801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.512610    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72" event={"ID":"d5d863f5-b05e-4922-9f70-015323992554","Type":"ContainerDied","Data":"1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d"}
Oct 30 21:55:23.512801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.512732    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72" event={"ID":"d5d863f5-b05e-4922-9f70-015323992554","Type":"ContainerDied","Data":"6e602f1813f984e690620b45226e765ad914e51cf9fcfbc81afe4c690df8f6d2"}
Oct 30 21:55:23.512801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.512763    4633 scope.go:122] "RemoveContainer" containerID="1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d"
Oct 30 21:55:23.513184 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.513133    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"
Oct 30 21:55:23.517389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.517187    4633 generic.go:356] "Generic (PLEG): container finished" podID="26abf544-f44b-43da-9cbf-50968294c4dc" containerID="c6ed724bd4f6ea3abec644aa8056d034487cc465b01f3c010d9e43e502bb8544" exitCode=0
Oct 30 21:55:23.517389 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.517318    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" event={"ID":"26abf544-f44b-43da-9cbf-50968294c4dc","Type":"ContainerDied","Data":"c6ed724bd4f6ea3abec644aa8056d034487cc465b01f3c010d9e43e502bb8544"}
Oct 30 21:55:23.521033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.519706    4633 generic.go:356] "Generic (PLEG): container finished" podID="9645737c-61cc-475d-9832-d5dda1b7dee9" containerID="fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948" exitCode=0
Oct 30 21:55:23.521033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.519861    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2" event={"ID":"9645737c-61cc-475d-9832-d5dda1b7dee9","Type":"ContainerDied","Data":"fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948"}
Oct 30 21:55:23.521033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.519886    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2" event={"ID":"9645737c-61cc-475d-9832-d5dda1b7dee9","Type":"ContainerDied","Data":"2701fdda90e910a32b34917e64e42e38cd48bb3477bee2b33c0067749aa1517c"}
Oct 30 21:55:23.521033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.519945    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"
Oct 30 21:55:23.529903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.529888    4633 scope.go:122] "RemoveContainer" containerID="1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d"
Oct 30 21:55:23.530200 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:23.530170    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d\": container with ID starting with 1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d not found: ID does not exist" containerID="1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d"
Oct 30 21:55:23.530308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.530211    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d"} err="failed to get container status \"1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d\": rpc error: code = NotFound desc = could not find container \"1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d\": container with ID starting with 1f04bc03a8801b8de517ec6ef454f31fbada4791df483c831418ac239e00142d not found: ID does not exist"
Oct 30 21:55:23.530308 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.530238    4633 scope.go:122] "RemoveContainer" containerID="88f9b54d330fd5a5e679081790c7797637ae0347a7eb6ce6160994c2c410239d"
Oct 30 21:55:23.546425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.546398    4633 scope.go:122] "RemoveContainer" containerID="fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948"
Oct 30 21:55:23.558853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.558829    4633 scope.go:122] "RemoveContainer" containerID="fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948"
Oct 30 21:55:23.559251 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:23.559220    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948\": container with ID starting with fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948 not found: ID does not exist" containerID="fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948"
Oct 30 21:55:23.559378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.559264    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948"} err="failed to get container status \"fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948\": rpc error: code = NotFound desc = could not find container \"fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948\": container with ID starting with fd43036e332a10bc37a62aba609d2b8b236cffed6dbde7764e1f27a14a3c1948 not found: ID does not exist"
Oct 30 21:55:23.559957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.559938    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"]
Oct 30 21:55:23.569384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.569359    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:55:23.570357 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.570335    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-726a18-77975cbd99-4xp72"]
Oct 30 21:55:23.585462 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.585438    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"]
Oct 30 21:55:23.617435 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.617401    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e8d315/test-model-e8d315-kserve-556d99995-qh6v2"]
Oct 30 21:55:23.686407 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686374    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-envoy\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.686602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686413    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-certs\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.686602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686510    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-socket\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.686602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686582    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-kube-api-access-5hsgb\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-kube-api-access-5hsgb\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.686825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686609    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-istio-token\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-istio-token\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.686825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686641    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26abf544-f44b-43da-9cbf-50968294c4dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26abf544-f44b-43da-9cbf-50968294c4dc-istiod-ca-cert\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.686825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686733    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-data\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.686825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686759    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26abf544-f44b-43da-9cbf-50968294c4dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26abf544-f44b-43da-9cbf-50968294c4dc-istio-podinfo\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.686825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686793    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-certs" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.687088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.686803    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-credential-socket\") pod \"26abf544-f44b-43da-9cbf-50968294c4dc\" (UID: \"26abf544-f44b-43da-9cbf-50968294c4dc\") "
Oct 30 21:55:23.687088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.687023    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-credential-socket" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.687258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.687125    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-socket" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.687258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.687149    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-data" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.687532 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.687506    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.687597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.687540    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.687597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.687556    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.687597 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.687571    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.687832 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.687811    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26abf544-f44b-43da-9cbf-50968294c4dc-istiod-ca-cert" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:23.689543 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.689512    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-envoy" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.689690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.689544    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-istio-token" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:23.689690 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.689623    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26abf544-f44b-43da-9cbf-50968294c4dc-istio-podinfo" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:23.690142 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.690117    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-kube-api-access-5hsgb" pod "26abf544-f44b-43da-9cbf-50968294c4dc" (UID: "26abf544-f44b-43da-9cbf-50968294c4dc"). InnerVolumeSpecName "kube-api-access-5hsgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:23.789456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.789370    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hsgb\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-kube-api-access-5hsgb\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.789456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.789411    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26abf544-f44b-43da-9cbf-50968294c4dc-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.789456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.789425    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26abf544-f44b-43da-9cbf-50968294c4dc-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.789456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.789439    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26abf544-f44b-43da-9cbf-50968294c4dc-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.789456 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.789454    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26abf544-f44b-43da-9cbf-50968294c4dc-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.792304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.792276    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9645737c-61cc-475d-9832-d5dda1b7dee9" path="/var/lib/kubelet/pods/9645737c-61cc-475d-9832-d5dda1b7dee9/volumes"
Oct 30 21:55:23.792764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:23.792748    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d863f5-b05e-4922-9f70-015323992554" path="/var/lib/kubelet/pods/d5d863f5-b05e-4922-9f70-015323992554/volumes"
Oct 30 21:55:24.531741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:24.531705    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"
Oct 30 21:55:24.532204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:24.531706    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj" event={"ID":"26abf544-f44b-43da-9cbf-50968294c4dc","Type":"ContainerDied","Data":"155fe516bae7c4ac429ce8e095470b884325ed548c66542f82df1c18091df96c"}
Oct 30 21:55:24.532204 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:24.531855    4633 scope.go:122] "RemoveContainer" containerID="c6ed724bd4f6ea3abec644aa8056d034487cc465b01f3c010d9e43e502bb8544"
Oct 30 21:55:24.557115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:24.557083    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"]
Oct 30 21:55:24.560615 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:24.560590    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-tnnpj"]
Oct 30 21:55:25.050148 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:25.050110    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:55:25.794702 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:25.794648    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" path="/var/lib/kubelet/pods/26abf544-f44b-43da-9cbf-50968294c4dc/volumes"
Oct 30 21:55:27.008932 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.008892    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"]
Oct 30 21:55:27.009439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.009143    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k" podUID="676be30a-0353-4bbf-8775-1219db088582" containerName="maas-api" containerID="cri-o://94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c" gracePeriod=30
Oct 30 21:55:27.371880 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.371855    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:55:27.464961 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.464926    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676be30a-0353-4bbf-8775-1219db088582-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676be30a-0353-4bbf-8775-1219db088582-openshift-service-ca\") pod \"676be30a-0353-4bbf-8775-1219db088582\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") "
Oct 30 21:55:27.465168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.464985    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-maas-api-tls\") pod \"676be30a-0353-4bbf-8775-1219db088582\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") "
Oct 30 21:55:27.465168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.465019    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-metrics-certs\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-metrics-certs\") pod \"676be30a-0353-4bbf-8775-1219db088582\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") "
Oct 30 21:55:27.465168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.465160    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676be30a-0353-4bbf-8775-1219db088582-kube-api-access-8wg26\" (UniqueName: \"kubernetes.io/projected/676be30a-0353-4bbf-8775-1219db088582-kube-api-access-8wg26\") pod \"676be30a-0353-4bbf-8775-1219db088582\" (UID: \"676be30a-0353-4bbf-8775-1219db088582\") "
Oct 30 21:55:27.466169 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.466137    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676be30a-0353-4bbf-8775-1219db088582-openshift-service-ca" pod "676be30a-0353-4bbf-8775-1219db088582" (UID: "676be30a-0353-4bbf-8775-1219db088582"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:27.468165 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.468135    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-maas-api-tls" pod "676be30a-0353-4bbf-8775-1219db088582" (UID: "676be30a-0353-4bbf-8775-1219db088582"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:27.468579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.468554    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676be30a-0353-4bbf-8775-1219db088582-kube-api-access-8wg26" pod "676be30a-0353-4bbf-8775-1219db088582" (UID: "676be30a-0353-4bbf-8775-1219db088582"). InnerVolumeSpecName "kube-api-access-8wg26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:27.468968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.468948    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-metrics-certs" pod "676be30a-0353-4bbf-8775-1219db088582" (UID: "676be30a-0353-4bbf-8775-1219db088582"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:27.559602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.559562    4633 generic.go:356] "Generic (PLEG): container finished" podID="676be30a-0353-4bbf-8775-1219db088582" containerID="94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c" exitCode=0
Oct 30 21:55:27.559846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.559612    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"
Oct 30 21:55:27.559846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.559641    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k" event={"ID":"676be30a-0353-4bbf-8775-1219db088582","Type":"ContainerDied","Data":"94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c"}
Oct 30 21:55:27.559846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.559759    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k" event={"ID":"676be30a-0353-4bbf-8775-1219db088582","Type":"ContainerDied","Data":"57d7e2be68c10f1ed2ba646cf0d33f0f42c51969e66af40c1c4a08b571d3c096"}
Oct 30 21:55:27.559846 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.559792    4633 scope.go:122] "RemoveContainer" containerID="94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c"
Oct 30 21:55:27.566811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.566787    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wg26\" (UniqueName: \"kubernetes.io/projected/676be30a-0353-4bbf-8775-1219db088582-kube-api-access-8wg26\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:27.566811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.566812    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/676be30a-0353-4bbf-8775-1219db088582-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:27.566957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.566823    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:27.566957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.566832    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/676be30a-0353-4bbf-8775-1219db088582-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:27.572973 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.572947    4633 scope.go:122] "RemoveContainer" containerID="94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c"
Oct 30 21:55:27.573321 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:27.573301    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c\": container with ID starting with 94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c not found: ID does not exist" containerID="94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c"
Oct 30 21:55:27.573417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.573330    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c"} err="failed to get container status \"94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c\": rpc error: code = NotFound desc = could not find container \"94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c\": container with ID starting with 94f794002daada9d53c45b3aa808fa17a41092e98f86ccc6b956f22f9bc72f1c not found: ID does not exist"
Oct 30 21:55:27.585186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.585155    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"]
Oct 30 21:55:27.588889 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.588865    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-adf308-6d97f76dcb-6kz6k"]
Oct 30 21:55:27.792879 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:27.792793    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676be30a-0353-4bbf-8775-1219db088582" path="/var/lib/kubelet/pods/676be30a-0353-4bbf-8775-1219db088582/volumes"
Oct 30 21:55:31.599715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:31.599683    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"]
Oct 30 21:55:31.600209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:31.599878    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy" containerID="cri-o://866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8" gracePeriod=30
Oct 30 21:55:33.151197 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.151152    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"]
Oct 30 21:55:33.151628 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.151407    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p" podUID="0a3ac7f8-6268-40e8-9599-5c11cca9390d" containerName="maas-api" containerID="cri-o://33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959" gracePeriod=30
Oct 30 21:55:33.542547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.542525    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:55:33.608477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.608435    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3ac7f8-6268-40e8-9599-5c11cca9390d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ac7f8-6268-40e8-9599-5c11cca9390d-openshift-service-ca\") pod \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") "
Oct 30 21:55:33.608649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.608550    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-maas-api-tls\") pod \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") "
Oct 30 21:55:33.608649 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.608605    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-metrics-certs\") pod \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") "
Oct 30 21:55:33.608856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.608650    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3ac7f8-6268-40e8-9599-5c11cca9390d-kube-api-access-nvpzm\" (UniqueName: \"kubernetes.io/projected/0a3ac7f8-6268-40e8-9599-5c11cca9390d-kube-api-access-nvpzm\") pod \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\" (UID: \"0a3ac7f8-6268-40e8-9599-5c11cca9390d\") "
Oct 30 21:55:33.608856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.608808    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3ac7f8-6268-40e8-9599-5c11cca9390d-openshift-service-ca" pod "0a3ac7f8-6268-40e8-9599-5c11cca9390d" (UID: "0a3ac7f8-6268-40e8-9599-5c11cca9390d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:33.609034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.609009    4633 generic.go:356] "Generic (PLEG): container finished" podID="0a3ac7f8-6268-40e8-9599-5c11cca9390d" containerID="33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959" exitCode=0
Oct 30 21:55:33.609253 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.609234    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p" event={"ID":"0a3ac7f8-6268-40e8-9599-5c11cca9390d","Type":"ContainerDied","Data":"33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959"}
Oct 30 21:55:33.609320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.609278    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ac7f8-6268-40e8-9599-5c11cca9390d-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:33.609320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.609289    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p" event={"ID":"0a3ac7f8-6268-40e8-9599-5c11cca9390d","Type":"ContainerDied","Data":"69cf4364e68eebac29aa329f42ed769b16baf379a972bee7c54df1818eda0d99"}
Oct 30 21:55:33.609428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.609325    4633 scope.go:122] "RemoveContainer" containerID="33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959"
Oct 30 21:55:33.609489 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.609476    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"
Oct 30 21:55:33.613088 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.613062    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3ac7f8-6268-40e8-9599-5c11cca9390d-kube-api-access-nvpzm" pod "0a3ac7f8-6268-40e8-9599-5c11cca9390d" (UID: "0a3ac7f8-6268-40e8-9599-5c11cca9390d"). InnerVolumeSpecName "kube-api-access-nvpzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:33.614547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.614517    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-maas-api-tls" pod "0a3ac7f8-6268-40e8-9599-5c11cca9390d" (UID: "0a3ac7f8-6268-40e8-9599-5c11cca9390d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:33.614705 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.614597    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-metrics-certs" pod "0a3ac7f8-6268-40e8-9599-5c11cca9390d" (UID: "0a3ac7f8-6268-40e8-9599-5c11cca9390d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:33.642531 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.642507    4633 scope.go:122] "RemoveContainer" containerID="33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959"
Oct 30 21:55:33.642900 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:33.642882    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959\": container with ID starting with 33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959 not found: ID does not exist" containerID="33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959"
Oct 30 21:55:33.642967 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.642908    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959"} err="failed to get container status \"33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959\": rpc error: code = NotFound desc = could not find container \"33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959\": container with ID starting with 33ecafced4900346f7450e50483fa22e5bcf5da12949dc3038b00775a87df959 not found: ID does not exist"
Oct 30 21:55:33.734737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.733538    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:33.734737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.733571    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a3ac7f8-6268-40e8-9599-5c11cca9390d-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:33.734737 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.733587    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvpzm\" (UniqueName: \"kubernetes.io/projected/0a3ac7f8-6268-40e8-9599-5c11cca9390d-kube-api-access-nvpzm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:33.930677 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.930640    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"]
Oct 30 21:55:33.935402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:33.935373    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4a0a5-7cbb9659fb-tsr8p"]
Oct 30 21:55:35.657567 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:35.657529    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:55:35.795075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:35.795036    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3ac7f8-6268-40e8-9599-5c11cca9390d" path="/var/lib/kubelet/pods/0a3ac7f8-6268-40e8-9599-5c11cca9390d/volumes"
Oct 30 21:55:36.329704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:36.329661    4633 scope.go:122] "RemoveContainer" containerID="849dd05bd0563df394be52032da0815bd1557af6b2d49178d7acfffe9e8d4c8d"
Oct 30 21:55:36.882716 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:36.882652    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:55:37.014523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014493    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istiod-ca-cert\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.014727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014538    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-certs\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.014727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014600    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-socket\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.014823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014731    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-kube-api-access-sbf88\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-kube-api-access-sbf88\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.014823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014771    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-credential-socket\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.014899 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014833    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-token\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-token\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.014952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014897    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-podinfo\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.014952 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014926    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-envoy\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.015066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014888    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istiod-ca-cert" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:37.015066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014912    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-certs" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.015066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014940    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-socket" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.015066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.014953    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-data\") pod \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\" (UID: \"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a\") "
Oct 30 21:55:37.015286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.015214    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-data" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.015286 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.015236    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-credential-socket" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.015612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.015586    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.015734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.015617    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.015734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.015632    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.015734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.015645    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.015734 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.015659    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.017148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.017127    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-podinfo" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:37.017355 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.017330    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-kube-api-access-sbf88" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "kube-api-access-sbf88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:37.017720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.017693    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-envoy" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.017831 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.017715    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-token" pod "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" (UID: "036bc02a-67d6-4d8e-86b6-44fba4ad6f1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:37.116703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.116654    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbf88\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-kube-api-access-sbf88\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.116703 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.116707    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.116892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.116719    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.116892 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.116728    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.680484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.680445    4633 generic.go:356] "Generic (PLEG): container finished" podID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerID="866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8" exitCode=0
Oct 30 21:55:37.680949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.680923    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" event={"ID":"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a","Type":"ContainerDied","Data":"866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8"}
Oct 30 21:55:37.681193 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.681174    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp" event={"ID":"036bc02a-67d6-4d8e-86b6-44fba4ad6f1a","Type":"ContainerDied","Data":"0c4c2b8ab0c4993763c375442fc53493bffb11551a9df8fe4b49e37996d5edcf"}
Oct 30 21:55:37.681317 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.681304    4633 scope.go:122] "RemoveContainer" containerID="866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8"
Oct 30 21:55:37.681560 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.681543    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"
Oct 30 21:55:37.720264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.720220    4633 scope.go:122] "RemoveContainer" containerID="866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8"
Oct 30 21:55:37.720878 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:37.720800    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8\": container with ID starting with 866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8 not found: ID does not exist" containerID="866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8"
Oct 30 21:55:37.720878 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.720844    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8"} err="failed to get container status \"866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8\": rpc error: code = NotFound desc = could not find container \"866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8\": container with ID starting with 866015eb26bd597d54e538c53f3090907e1c734a453bdd7b9e1b5b93ecb5cbb8 not found: ID does not exist"
Oct 30 21:55:37.743825 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.743734    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"]
Oct 30 21:55:37.749366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.749336    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-ml8xp"]
Oct 30 21:55:37.794590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:37.794559    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" path="/var/lib/kubelet/pods/036bc02a-67d6-4d8e-86b6-44fba4ad6f1a/volumes"
Oct 30 21:55:41.103612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:41.103571    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"]
Oct 30 21:55:41.104164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:41.103791    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy" containerID="cri-o://db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2" gracePeriod=30
Oct 30 21:55:44.210210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:44.210171    4633 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:55:44.210746 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:44.210241    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:55:46.441264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.441238    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:55:46.552909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.552876    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-data\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.552936    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-certs\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.552959    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-envoy\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553041    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-podinfo\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553060    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istiod-ca-cert\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553091    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-socket\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553202    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-data" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.553419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553230    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-certs" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.553419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553250    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-kube-api-access-frqvf\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-kube-api-access-frqvf\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553287    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-token\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-token\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553419 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553333    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-credential-socket\") pod \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\" (UID: \"3ff05b49-76c1-45dc-9f4d-3d07699c12d7\") "
Oct 30 21:55:46.553757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553452    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-socket" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.553757 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553559    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istiod-ca-cert" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:46.554064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553794    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-credential-socket" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.554064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553919    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.554064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553932    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.554064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553942    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.554064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553953    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.554064 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.553966    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.555622 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.555598    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-envoy" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.555822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.555649    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-kube-api-access-frqvf" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "kube-api-access-frqvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:46.555977 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.555954    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-token" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:46.556060 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.555988    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-podinfo" pod "3ff05b49-76c1-45dc-9f4d-3d07699c12d7" (UID: "3ff05b49-76c1-45dc-9f4d-3d07699c12d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:46.655570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.655505    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.655570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.655560    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.655570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.655575    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqvf\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-kube-api-access-frqvf\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.655570 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.655588    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ff05b49-76c1-45dc-9f4d-3d07699c12d7-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.809033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.808939    4633 generic.go:356] "Generic (PLEG): container finished" podID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerID="db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2" exitCode=0
Oct 30 21:55:46.809033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.808992    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"
Oct 30 21:55:46.809033 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.809022    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" event={"ID":"3ff05b49-76c1-45dc-9f4d-3d07699c12d7","Type":"ContainerDied","Data":"db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2"}
Oct 30 21:55:46.809347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.809078    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp" event={"ID":"3ff05b49-76c1-45dc-9f4d-3d07699c12d7","Type":"ContainerDied","Data":"d19d91277c40d03e40a27499f67e0d9fc075a2f302cf443fa6cc7949e182a0d7"}
Oct 30 21:55:46.809347 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.809102    4633 scope.go:122] "RemoveContainer" containerID="db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2"
Oct 30 21:55:46.821561 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.821541    4633 scope.go:122] "RemoveContainer" containerID="ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901"
Oct 30 21:55:46.836339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.833689    4633 scope.go:122] "RemoveContainer" containerID="db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2"
Oct 30 21:55:46.836339 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:46.834171    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2\": container with ID starting with db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2 not found: ID does not exist" containerID="db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2"
Oct 30 21:55:46.836339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.834207    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2"} err="failed to get container status \"db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2\": rpc error: code = NotFound desc = could not find container \"db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2\": container with ID starting with db8a5fbcb5e811b22846720bfe9d75c803f31af21e06d30f10c5916b0ecdb6d2 not found: ID does not exist"
Oct 30 21:55:46.836339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.834232    4633 scope.go:122] "RemoveContainer" containerID="ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901"
Oct 30 21:55:46.836339 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:46.834731    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901\": container with ID starting with ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901 not found: ID does not exist" containerID="ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901"
Oct 30 21:55:46.836339 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.834762    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901"} err="failed to get container status \"ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901\": rpc error: code = NotFound desc = could not find container \"ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901\": container with ID starting with ccffaf760af3e45481f7c48b9a9eac022cf5fe8c0cce1ce807546a77949db901 not found: ID does not exist"
Oct 30 21:55:46.842312 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.842283    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"]
Oct 30 21:55:46.849364 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:46.849338    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-6p6bp"]
Oct 30 21:55:47.792073 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:47.792041    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" path="/var/lib/kubelet/pods/3ff05b49-76c1-45dc-9f4d-3d07699c12d7/volumes"
Oct 30 21:55:51.656808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:51.656771    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"]
Oct 30 21:55:51.657304 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:51.657047    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp" podUID="af90c02b-8a15-411b-bbcb-d3d658effbec" containerName="maas-api" containerID="cri-o://8c810d75fc8299393768de5c4376e53b54a31587111c352f80914354d3a1b08b" gracePeriod=30
Oct 30 21:55:51.858642 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:51.858605    4633 generic.go:356] "Generic (PLEG): container finished" podID="af90c02b-8a15-411b-bbcb-d3d658effbec" containerID="8c810d75fc8299393768de5c4376e53b54a31587111c352f80914354d3a1b08b" exitCode=0
Oct 30 21:55:51.860733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:51.858889    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp" event={"ID":"af90c02b-8a15-411b-bbcb-d3d658effbec","Type":"ContainerDied","Data":"8c810d75fc8299393768de5c4376e53b54a31587111c352f80914354d3a1b08b"}
Oct 30 21:55:52.075217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.075186    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:55:52.247877 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.247844    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-maas-api-tls\") pod \"af90c02b-8a15-411b-bbcb-d3d658effbec\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") "
Oct 30 21:55:52.248038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.247921    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af90c02b-8a15-411b-bbcb-d3d658effbec-kube-api-access-9ktjl\" (UniqueName: \"kubernetes.io/projected/af90c02b-8a15-411b-bbcb-d3d658effbec-kube-api-access-9ktjl\") pod \"af90c02b-8a15-411b-bbcb-d3d658effbec\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") "
Oct 30 21:55:52.248038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.247960    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-metrics-certs\") pod \"af90c02b-8a15-411b-bbcb-d3d658effbec\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") "
Oct 30 21:55:52.248038 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.247982    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af90c02b-8a15-411b-bbcb-d3d658effbec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af90c02b-8a15-411b-bbcb-d3d658effbec-openshift-service-ca\") pod \"af90c02b-8a15-411b-bbcb-d3d658effbec\" (UID: \"af90c02b-8a15-411b-bbcb-d3d658effbec\") "
Oct 30 21:55:52.248446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.248409    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af90c02b-8a15-411b-bbcb-d3d658effbec-openshift-service-ca" pod "af90c02b-8a15-411b-bbcb-d3d658effbec" (UID: "af90c02b-8a15-411b-bbcb-d3d658effbec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:52.248602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.248581    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af90c02b-8a15-411b-bbcb-d3d658effbec-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.250471 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.250446    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-metrics-certs" pod "af90c02b-8a15-411b-bbcb-d3d658effbec" (UID: "af90c02b-8a15-411b-bbcb-d3d658effbec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:52.250579 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.250481    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-maas-api-tls" pod "af90c02b-8a15-411b-bbcb-d3d658effbec" (UID: "af90c02b-8a15-411b-bbcb-d3d658effbec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:52.250648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.250625    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af90c02b-8a15-411b-bbcb-d3d658effbec-kube-api-access-9ktjl" pod "af90c02b-8a15-411b-bbcb-d3d658effbec" (UID: "af90c02b-8a15-411b-bbcb-d3d658effbec"). InnerVolumeSpecName "kube-api-access-9ktjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:52.351752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.350660    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.351752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.350727    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af90c02b-8a15-411b-bbcb-d3d658effbec-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.351752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.350744    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ktjl\" (UniqueName: \"kubernetes.io/projected/af90c02b-8a15-411b-bbcb-d3d658effbec-kube-api-access-9ktjl\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.866581 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.866550    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"
Oct 30 21:55:52.867085 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.866556    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp" event={"ID":"af90c02b-8a15-411b-bbcb-d3d658effbec","Type":"ContainerDied","Data":"19a44763694418dfee0e7ac8e91d14f0c7ec8eb66b35f79c8666655b1d1e6eaf"}
Oct 30 21:55:52.867085 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.866702    4633 scope.go:122] "RemoveContainer" containerID="8c810d75fc8299393768de5c4376e53b54a31587111c352f80914354d3a1b08b"
Oct 30 21:55:52.901687 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.901634    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"]
Oct 30 21:55:52.905049 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:52.905024    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-904993-7bc6c5d74f-bhqgp"]
Oct 30 21:55:53.793134 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:53.793095    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af90c02b-8a15-411b-bbcb-d3d658effbec" path="/var/lib/kubelet/pods/af90c02b-8a15-411b-bbcb-d3d658effbec/volumes"
Oct 30 21:55:53.817862 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:55:53.817828    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:55:55.015160 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:55:55.015119    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:56:03.092682 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:03.092629    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"]
Oct 30 21:56:03.093153 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:03.092895    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy" containerID="cri-o://a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27" gracePeriod=30
Oct 30 21:56:08.383796 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.383714    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.437651    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-podinfo\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.437765    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-socket\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.437843    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510937e8-5b5e-474d-9d5b-452eb5062cb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510937e8-5b5e-474d-9d5b-452eb5062cb8-istiod-ca-cert\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.437903    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-certs\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.437925    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-data\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.437970    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-kube-api-access-hvbb6\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-kube-api-access-hvbb6\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.437991    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-credential-socket\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.438032    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-envoy\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.438994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.438060    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-token\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-token\") pod \"510937e8-5b5e-474d-9d5b-452eb5062cb8\" (UID: \"510937e8-5b5e-474d-9d5b-452eb5062cb8\") "
Oct 30 21:56:08.439733 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.439627    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-certs" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.440756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.440705    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-data" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.440756 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.440583    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510937e8-5b5e-474d-9d5b-452eb5062cb8-istiod-ca-cert" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:08.441093 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.440917    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-credential-socket" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.441425 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.441390    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-socket" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.442499 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.442470    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-token" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:08.444007 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.443982    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-podinfo" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:08.450262 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.450234    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-kube-api-access-hvbb6" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "kube-api-access-hvbb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:08.451441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.451412    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-envoy" pod "510937e8-5b5e-474d-9d5b-452eb5062cb8" (UID: "510937e8-5b5e-474d-9d5b-452eb5062cb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.539623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539586    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.539623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539618    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510937e8-5b5e-474d-9d5b-452eb5062cb8-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.539623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539628    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.539918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539637    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.539918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539646    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvbb6\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-kube-api-access-hvbb6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.539918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539655    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.539918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539663    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.539918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539708    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.539918 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:08.539717    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510937e8-5b5e-474d-9d5b-452eb5062cb8-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:09.054140 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.054108    4633 generic.go:356] "Generic (PLEG): container finished" podID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerID="a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27" exitCode=0
Oct 30 21:56:09.054341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.054163    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"
Oct 30 21:56:09.054341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.054189    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" event={"ID":"510937e8-5b5e-474d-9d5b-452eb5062cb8","Type":"ContainerDied","Data":"a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27"}
Oct 30 21:56:09.054341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.054255    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl" event={"ID":"510937e8-5b5e-474d-9d5b-452eb5062cb8","Type":"ContainerDied","Data":"5448b56912733f6a680f8b7e799fc72ab0df467eac4ba1a72d59e47f3a10ea20"}
Oct 30 21:56:09.054341 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.054286    4633 scope.go:122] "RemoveContainer" containerID="a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27"
Oct 30 21:56:09.067320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.067296    4633 scope.go:122] "RemoveContainer" containerID="75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec"
Oct 30 21:56:09.078244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.078222    4633 scope.go:122] "RemoveContainer" containerID="a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27"
Oct 30 21:56:09.078554 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:09.078526    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27\": container with ID starting with a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27 not found: ID does not exist" containerID="a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27"
Oct 30 21:56:09.078627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.078561    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27"} err="failed to get container status \"a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27\": rpc error: code = NotFound desc = could not find container \"a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27\": container with ID starting with a2d69fcaa4a559a8d3259000bc46a28657a13bd7438caade62222a15c643be27 not found: ID does not exist"
Oct 30 21:56:09.078627 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.078582    4633 scope.go:122] "RemoveContainer" containerID="75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec"
Oct 30 21:56:09.078934 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:09.078918    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec\": container with ID starting with 75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec not found: ID does not exist" containerID="75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec"
Oct 30 21:56:09.078983 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.078945    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec"} err="failed to get container status \"75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec\": rpc error: code = NotFound desc = could not find container \"75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec\": container with ID starting with 75daf10dea00fc8586ac43e9ecbabb10a97de5d6316e52019d36d2b08a87d2ec not found: ID does not exist"
Oct 30 21:56:09.084076 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.084052    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"]
Oct 30 21:56:09.090630 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.090603    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-p9tpl"]
Oct 30 21:56:09.792066 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.792029    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" path="/var/lib/kubelet/pods/510937e8-5b5e-474d-9d5b-452eb5062cb8/volumes"
Oct 30 21:56:09.875330 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.875291    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"]
Oct 30 21:56:09.875547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:09.875503    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" containerID="cri-o://7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2" gracePeriod=30
Oct 30 21:56:10.700218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:10.700185    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"]
Oct 30 21:56:10.700450 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:10.700413    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp" podUID="f4238ff6-1550-434d-979b-f44b9bd50a99" containerName="maas-api" containerID="cri-o://b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162" gracePeriod=30
Oct 30 21:56:11.020659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.020632    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:56:11.139908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.139873    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4238ff6-1550-434d-979b-f44b9bd50a99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4238ff6-1550-434d-979b-f44b9bd50a99-openshift-service-ca\") pod \"f4238ff6-1550-434d-979b-f44b9bd50a99\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") "
Oct 30 21:56:11.140090 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.140055    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-metrics-certs\") pod \"f4238ff6-1550-434d-979b-f44b9bd50a99\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") "
Oct 30 21:56:11.140137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.140088    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4238ff6-1550-434d-979b-f44b9bd50a99-kube-api-access-8plqt\" (UniqueName: \"kubernetes.io/projected/f4238ff6-1550-434d-979b-f44b9bd50a99-kube-api-access-8plqt\") pod \"f4238ff6-1550-434d-979b-f44b9bd50a99\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") "
Oct 30 21:56:11.140195 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.140179    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-maas-api-tls\") pod \"f4238ff6-1550-434d-979b-f44b9bd50a99\" (UID: \"f4238ff6-1550-434d-979b-f44b9bd50a99\") "
Oct 30 21:56:11.143662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.143620    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-metrics-certs" pod "f4238ff6-1550-434d-979b-f44b9bd50a99" (UID: "f4238ff6-1550-434d-979b-f44b9bd50a99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:11.144325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.144296    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-maas-api-tls" pod "f4238ff6-1550-434d-979b-f44b9bd50a99" (UID: "f4238ff6-1550-434d-979b-f44b9bd50a99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:11.144968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.144941    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4238ff6-1550-434d-979b-f44b9bd50a99-openshift-service-ca" pod "f4238ff6-1550-434d-979b-f44b9bd50a99" (UID: "f4238ff6-1550-434d-979b-f44b9bd50a99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:11.150214 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.150189    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4238ff6-1550-434d-979b-f44b9bd50a99-kube-api-access-8plqt" pod "f4238ff6-1550-434d-979b-f44b9bd50a99" (UID: "f4238ff6-1550-434d-979b-f44b9bd50a99"). InnerVolumeSpecName "kube-api-access-8plqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:11.150508 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.150480    4633 generic.go:356] "Generic (PLEG): container finished" podID="f4238ff6-1550-434d-979b-f44b9bd50a99" containerID="b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162" exitCode=0
Oct 30 21:56:11.150631 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.150608    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp" event={"ID":"f4238ff6-1550-434d-979b-f44b9bd50a99","Type":"ContainerDied","Data":"b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162"}
Oct 30 21:56:11.150769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.150704    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp" event={"ID":"f4238ff6-1550-434d-979b-f44b9bd50a99","Type":"ContainerDied","Data":"d7c2215cf0f86c3c578b49a633b8b7d72bee82905b9876874f11f840ba138e86"}
Oct 30 21:56:11.150769 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.150742    4633 scope.go:122] "RemoveContainer" containerID="b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162"
Oct 30 21:56:11.150919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.150905    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"
Oct 30 21:56:11.176927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.176810    4633 scope.go:122] "RemoveContainer" containerID="b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162"
Oct 30 21:56:11.177247 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:11.177216    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162\": container with ID starting with b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162 not found: ID does not exist" containerID="b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162"
Oct 30 21:56:11.177318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.177260    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162"} err="failed to get container status \"b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162\": rpc error: code = NotFound desc = could not find container \"b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162\": container with ID starting with b6822522eb234ebad059202b72fdf8affbc2ccb4b54db8941049ce0a2e127162 not found: ID does not exist"
Oct 30 21:56:11.194130 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.194091    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"]
Oct 30 21:56:11.202747 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.202709    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-bfc50e-77df548895-xvhlp"]
Oct 30 21:56:11.241953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.241877    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:11.241953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.241905    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8plqt\" (UniqueName: \"kubernetes.io/projected/f4238ff6-1550-434d-979b-f44b9bd50a99-kube-api-access-8plqt\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:11.241953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.241925    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4238ff6-1550-434d-979b-f44b9bd50a99-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:11.241953 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.241935    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4238ff6-1550-434d-979b-f44b9bd50a99-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:11.817695 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:11.806011    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4238ff6-1550-434d-979b-f44b9bd50a99" path="/var/lib/kubelet/pods/f4238ff6-1550-434d-979b-f44b9bd50a99/volumes"
Oct 30 21:56:12.390223 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:12.390131    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"]
Oct 30 21:56:12.399250 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:12.399220    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"]
Oct 30 21:56:13.223563 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:13.223530    4633 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:56:13.223817 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:13.223595    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:56:15.180384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.180360    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:56:15.201645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.201607    4633 generic.go:356] "Generic (PLEG): container finished" podID="6592f635-0310-4fae-ad00-dec841dec9c7" containerID="7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2" exitCode=0
Oct 30 21:56:15.201850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.201647    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"
Oct 30 21:56:15.201850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.201722    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" event={"ID":"6592f635-0310-4fae-ad00-dec841dec9c7","Type":"ContainerDied","Data":"7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2"}
Oct 30 21:56:15.201850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.201790    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf" event={"ID":"6592f635-0310-4fae-ad00-dec841dec9c7","Type":"ContainerDied","Data":"8fc000e4d7831c6506fc55fc4a17cee1f24285b2217abacd2a76f9bb7c0c841b"}
Oct 30 21:56:15.201850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.201825    4633 scope.go:122] "RemoveContainer" containerID="7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2"
Oct 30 21:56:15.216231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.215575    4633 scope.go:122] "RemoveContainer" containerID="7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2"
Oct 30 21:56:15.216231 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:15.216027    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2\": container with ID starting with 7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2 not found: ID does not exist" containerID="7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2"
Oct 30 21:56:15.216231 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.216063    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2"} err="failed to get container status \"7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2\": rpc error: code = NotFound desc = could not find container \"7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2\": container with ID starting with 7f8fa8e7fe85afba3c9b34b6074ffd67f263d956f62f6077f87db91c5ee145c2 not found: ID does not exist"
Oct 30 21:56:15.302258 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302221    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-credential-socket\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302274    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6592f635-0310-4fae-ad00-dec841dec9c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592f635-0310-4fae-ad00-dec841dec9c7-istio-podinfo\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302301    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-data\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302333    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-kube-api-access-76k4q\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-kube-api-access-76k4q\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302426    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-socket\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302460    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-istio-token\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-istio-token\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302526    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-envoy\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302557    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-credential-socket" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.302970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302639    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-certs\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302638    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-data" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.302970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302702    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6592f635-0310-4fae-ad00-dec841dec9c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592f635-0310-4fae-ad00-dec841dec9c7-istiod-ca-cert\") pod \"6592f635-0310-4fae-ad00-dec841dec9c7\" (UID: \"6592f635-0310-4fae-ad00-dec841dec9c7\") "
Oct 30 21:56:15.302970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302829    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-socket" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.302970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.302914    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-certs" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.303475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.303276    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.303475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.303297    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.303475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.303307    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.303475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.303315    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.303475 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.303283    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6592f635-0310-4fae-ad00-dec841dec9c7-istiod-ca-cert" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:15.304890 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.304864    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6592f635-0310-4fae-ad00-dec841dec9c7-istio-podinfo" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:15.305010 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.304988    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-istio-token" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:15.305379 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.305361    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-envoy" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.305421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.305385    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-kube-api-access-76k4q" pod "6592f635-0310-4fae-ad00-dec841dec9c7" (UID: "6592f635-0310-4fae-ad00-dec841dec9c7"). InnerVolumeSpecName "kube-api-access-76k4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:15.404314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.404231    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6592f635-0310-4fae-ad00-dec841dec9c7-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.404314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.404264    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6592f635-0310-4fae-ad00-dec841dec9c7-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.404314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.404274    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76k4q\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-kube-api-access-76k4q\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.404314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.404285    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6592f635-0310-4fae-ad00-dec841dec9c7-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.404314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.404293    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6592f635-0310-4fae-ad00-dec841dec9c7-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.542764 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.542728    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"]
Oct 30 21:56:15.561092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.561058    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-4bdwf"]
Oct 30 21:56:15.791869 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:15.791783    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" path="/var/lib/kubelet/pods/6592f635-0310-4fae-ad00-dec841dec9c7/volumes"
Oct 30 21:56:17.390720 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:17.390635    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf" podUID="fef2c07d-1cb3-4b21-a25f-a525adb9a011" containerName="payload-processing" containerID="cri-o://c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c" gracePeriod=25
Oct 30 21:56:17.400294 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:17.400255    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq" podUID="e482d2d9-f8db-4d89-b704-73aa74c6417c" containerName="payload-pre-processing" containerID="cri-o://0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b" gracePeriod=25
Oct 30 21:56:19.759819 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:19.759784    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"]
Oct 30 21:56:19.760266 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:19.760024    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z" podUID="c8f26023-41db-4a13-b6be-912de216ccc5" containerName="maas-api" containerID="cri-o://382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65" gracePeriod=30
Oct 30 21:56:20.085874 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.085530    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:56:20.239043 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.239007    4633 generic.go:356] "Generic (PLEG): container finished" podID="c8f26023-41db-4a13-b6be-912de216ccc5" containerID="382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65" exitCode=0
Oct 30 21:56:20.239244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.239057    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"
Oct 30 21:56:20.239244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.239066    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z" event={"ID":"c8f26023-41db-4a13-b6be-912de216ccc5","Type":"ContainerDied","Data":"382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65"}
Oct 30 21:56:20.239244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.239128    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z" event={"ID":"c8f26023-41db-4a13-b6be-912de216ccc5","Type":"ContainerDied","Data":"e8b826255fbf288bbaa77524c66dbc4230ddd3ff6bc414f53a1793358d412d9b"}
Oct 30 21:56:20.239244 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.239151    4633 scope.go:122] "RemoveContainer" containerID="382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65"
Oct 30 21:56:20.251024 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.251001    4633 scope.go:122] "RemoveContainer" containerID="382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65"
Oct 30 21:56:20.251353 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:20.251332    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65\": container with ID starting with 382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65 not found: ID does not exist" containerID="382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65"
Oct 30 21:56:20.251428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.251362    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65"} err="failed to get container status \"382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65\": rpc error: code = NotFound desc = could not find container \"382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65\": container with ID starting with 382c6982a4e8b09c346742c4d39f8b3a3005636c0a7a8117f58cb68d94ce8d65 not found: ID does not exist"
Oct 30 21:56:20.269147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.269074    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls\") pod \"c8f26023-41db-4a13-b6be-912de216ccc5\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") "
Oct 30 21:56:20.269147 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.269126    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f26023-41db-4a13-b6be-912de216ccc5-kube-api-access-fpkbs\" (UniqueName: \"kubernetes.io/projected/c8f26023-41db-4a13-b6be-912de216ccc5-kube-api-access-fpkbs\") pod \"c8f26023-41db-4a13-b6be-912de216ccc5\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") "
Oct 30 21:56:20.269366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.269178    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f26023-41db-4a13-b6be-912de216ccc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f26023-41db-4a13-b6be-912de216ccc5-openshift-service-ca\") pod \"c8f26023-41db-4a13-b6be-912de216ccc5\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") "
Oct 30 21:56:20.269366 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.269236    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-metrics-certs\") pod \"c8f26023-41db-4a13-b6be-912de216ccc5\" (UID: \"c8f26023-41db-4a13-b6be-912de216ccc5\") "
Oct 30 21:56:20.269542 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.269520    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f26023-41db-4a13-b6be-912de216ccc5-openshift-service-ca" pod "c8f26023-41db-4a13-b6be-912de216ccc5" (UID: "c8f26023-41db-4a13-b6be-912de216ccc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:20.269801 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.269784    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f26023-41db-4a13-b6be-912de216ccc5-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:20.271491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.271467    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls" pod "c8f26023-41db-4a13-b6be-912de216ccc5" (UID: "c8f26023-41db-4a13-b6be-912de216ccc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:20.271623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.271501    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f26023-41db-4a13-b6be-912de216ccc5-kube-api-access-fpkbs" pod "c8f26023-41db-4a13-b6be-912de216ccc5" (UID: "c8f26023-41db-4a13-b6be-912de216ccc5"). InnerVolumeSpecName "kube-api-access-fpkbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:20.271623 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.271550    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-metrics-certs" pod "c8f26023-41db-4a13-b6be-912de216ccc5" (UID: "c8f26023-41db-4a13-b6be-912de216ccc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:20.382566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.382482    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:20.382566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.382523    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkbs\" (UniqueName: \"kubernetes.io/projected/c8f26023-41db-4a13-b6be-912de216ccc5-kube-api-access-fpkbs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:20.382566 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.382539    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f26023-41db-4a13-b6be-912de216ccc5-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:20.574843 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.574808    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"]
Oct 30 21:56:20.591331 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:20.591289    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fb2b58-5dbcb6b6c5-twz7z"]
Oct 30 21:56:21.227876 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:21.227835    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:56:21.791697 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:21.791641    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f26023-41db-4a13-b6be-912de216ccc5" path="/var/lib/kubelet/pods/c8f26023-41db-4a13-b6be-912de216ccc5/volumes"
Oct 30 21:56:23.066080 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:23.066039    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq" podUID="e482d2d9-f8db-4d89-b704-73aa74c6417c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:23.068727 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:23.067451    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf" podUID="fef2c07d-1cb3-4b21-a25f-a525adb9a011" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:23.253731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:23.253694    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"]
Oct 30 21:56:23.254263 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:23.254229    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" containerID="cri-o://058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271" gracePeriod=30
Oct 30 21:56:24.878515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:24.878478    4633 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:56:24.878929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:24.878565    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:56:25.014008 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:25.013966    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:56:28.545602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.545545    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:56:28.602226 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602189    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-credential-socket\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602247    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-envoy\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602298    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-token\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-token\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602421 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602349    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-podinfo\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602425    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-data\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602459    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-kube-api-access-x9bx6\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-kube-api-access-x9bx6\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602496    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87731e5-07ec-4842-8e0a-f98200c5acf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87731e5-07ec-4842-8e0a-f98200c5acf3-istiod-ca-cert\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602512    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-credential-socket" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.602610 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602567    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-certs\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602912 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602654    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-socket\") pod \"d87731e5-07ec-4842-8e0a-f98200c5acf3\" (UID: \"d87731e5-07ec-4842-8e0a-f98200c5acf3\") "
Oct 30 21:56:28.602912 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.602766    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-data" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.603148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.603110    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-socket" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.603277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.603210    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.603277 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.603234    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.603415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.603325    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87731e5-07ec-4842-8e0a-f98200c5acf3-istiod-ca-cert" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:28.603480 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.603438    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-certs" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.604998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.604973    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-envoy" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.605428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.605407    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-podinfo" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:28.605507 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.605436    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-token" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:28.605599 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.605580    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-kube-api-access-x9bx6" pod "d87731e5-07ec-4842-8e0a-f98200c5acf3" (UID: "d87731e5-07ec-4842-8e0a-f98200c5acf3"). InnerVolumeSpecName "kube-api-access-x9bx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:28.704680 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.704643    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.704856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.704703    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bx6\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-kube-api-access-x9bx6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.704856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.704720    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87731e5-07ec-4842-8e0a-f98200c5acf3-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.704856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.704729    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.704856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.704739    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.704856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.704747    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.704856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:28.704755    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87731e5-07ec-4842-8e0a-f98200c5acf3-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:29.325901 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.325858    4633 generic.go:356] "Generic (PLEG): container finished" podID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerID="058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271" exitCode=0
Oct 30 21:56:29.326107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.325930    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"
Oct 30 21:56:29.326107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.325934    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" event={"ID":"d87731e5-07ec-4842-8e0a-f98200c5acf3","Type":"ContainerDied","Data":"058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271"}
Oct 30 21:56:29.326107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.326016    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8" event={"ID":"d87731e5-07ec-4842-8e0a-f98200c5acf3","Type":"ContainerDied","Data":"03c4e79bbfedcce073aabe3adef553ad37f48f4724a2915ab82a4da64d45f561"}
Oct 30 21:56:29.326107 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.326042    4633 scope.go:122] "RemoveContainer" containerID="058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271"
Oct 30 21:56:29.337925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.337907    4633 scope.go:122] "RemoveContainer" containerID="47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865"
Oct 30 21:56:29.347589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.347568    4633 scope.go:122] "RemoveContainer" containerID="058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271"
Oct 30 21:56:29.347860 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:29.347838    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271\": container with ID starting with 058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271 not found: ID does not exist" containerID="058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271"
Oct 30 21:56:29.347909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.347873    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271"} err="failed to get container status \"058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271\": rpc error: code = NotFound desc = could not find container \"058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271\": container with ID starting with 058776d1848e7a73830be55d77a6813ea4047c24b4b17502a05bef33800d1271 not found: ID does not exist"
Oct 30 21:56:29.347909 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.347893    4633 scope.go:122] "RemoveContainer" containerID="47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865"
Oct 30 21:56:29.348171 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:29.348154    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865\": container with ID starting with 47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865 not found: ID does not exist" containerID="47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865"
Oct 30 21:56:29.348230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.348176    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865"} err="failed to get container status \"47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865\": rpc error: code = NotFound desc = could not find container \"47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865\": container with ID starting with 47d7a00f22fc3abfbcc53969c89019ba384ea9b76f4881aaac6e2f48fde48865 not found: ID does not exist"
Oct 30 21:56:29.359390 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.359368    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"]
Oct 30 21:56:29.367295 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.367274    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-vctk8"]
Oct 30 21:56:29.790848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:29.790809    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" path="/var/lib/kubelet/pods/d87731e5-07ec-4842-8e0a-f98200c5acf3/volumes"
Oct 30 21:56:32.436699 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:32.436649    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"]
Oct 30 21:56:32.437179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:32.436879    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy" containerID="cri-o://0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164" gracePeriod=30
Oct 30 21:56:33.065299 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:33.065250    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq" podUID="e482d2d9-f8db-4d89-b704-73aa74c6417c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:33.066927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:33.066893    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf" podUID="fef2c07d-1cb3-4b21-a25f-a525adb9a011" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:33.431388 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:33.431350    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:56:33.440092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:33.440066    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amx4xz_6e527a0b-1fa4-402c-ab56-6dd27aeca974/extract/0.log"
Oct 30 21:56:33.449928 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:33.449902    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:56:33.460624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:33.460595    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4cz8g_e0130da1-910e-4623-834b-40b494b2309d/ovn-acl-logging/0.log"
Oct 30 21:56:35.482523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:35.482484    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"]
Oct 30 21:56:35.483092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:35.482778    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" containerID="cri-o://cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda" gracePeriod=30
Oct 30 21:56:37.708275 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.708247    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:56:37.730488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.729727    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef2c07d-1cb3-4b21-a25f-a525adb9a011-kube-api-access-m6lj4\" (UniqueName: \"kubernetes.io/projected/fef2c07d-1cb3-4b21-a25f-a525adb9a011-kube-api-access-m6lj4\") pod \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\" (UID: \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\") "
Oct 30 21:56:37.730488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.729818    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef2c07d-1cb3-4b21-a25f-a525adb9a011-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef2c07d-1cb3-4b21-a25f-a525adb9a011-plugins-config-volume\") pod \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\" (UID: \"fef2c07d-1cb3-4b21-a25f-a525adb9a011\") "
Oct 30 21:56:37.731230 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.730926    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef2c07d-1cb3-4b21-a25f-a525adb9a011-plugins-config-volume" pod "fef2c07d-1cb3-4b21-a25f-a525adb9a011" (UID: "fef2c07d-1cb3-4b21-a25f-a525adb9a011"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:37.740280 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.740244    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef2c07d-1cb3-4b21-a25f-a525adb9a011-kube-api-access-m6lj4" pod "fef2c07d-1cb3-4b21-a25f-a525adb9a011" (UID: "fef2c07d-1cb3-4b21-a25f-a525adb9a011"). InnerVolumeSpecName "kube-api-access-m6lj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:37.741112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.741087    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:56:37.756453 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.756427    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:56:37.830979 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.830949    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-envoy\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831000    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-token\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-token\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831052    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-podinfo\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831124    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e482d2d9-f8db-4d89-b704-73aa74c6417c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e482d2d9-f8db-4d89-b704-73aa74c6417c-plugins-config-volume\") pod \"e482d2d9-f8db-4d89-b704-73aa74c6417c\" (UID: \"e482d2d9-f8db-4d89-b704-73aa74c6417c\") "
Oct 30 21:56:37.831180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831149    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-certs\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831202    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-kube-api-access-n8psz\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-kube-api-access-n8psz\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831274    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-data\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831316    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-socket\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831362    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-credential-socket\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831414    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbdd9f9d-6b83-495e-804f-e85d4185d396-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdd9f9d-6b83-495e-804f-e85d4185d396-istiod-ca-cert\") pod \"bbdd9f9d-6b83-495e-804f-e85d4185d396\" (UID: \"bbdd9f9d-6b83-495e-804f-e85d4185d396\") "
Oct 30 21:56:37.831624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831460    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e482d2d9-f8db-4d89-b704-73aa74c6417c-kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/e482d2d9-f8db-4d89-b704-73aa74c6417c-kube-api-access-zswm2\") pod \"e482d2d9-f8db-4d89-b704-73aa74c6417c\" (UID: \"e482d2d9-f8db-4d89-b704-73aa74c6417c\") "
Oct 30 21:56:37.831624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831549    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-certs" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.831856 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831808    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-socket" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.831919 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831871    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-credential-socket" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.832012 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.831989    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef2c07d-1cb3-4b21-a25f-a525adb9a011-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.832188 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.832168    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.832325 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.832311    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.832442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.832423    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6lj4\" (UniqueName: \"kubernetes.io/projected/fef2c07d-1cb3-4b21-a25f-a525adb9a011-kube-api-access-m6lj4\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.832548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.832203    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbdd9f9d-6b83-495e-804f-e85d4185d396-istiod-ca-cert" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:37.832645 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.832381    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e482d2d9-f8db-4d89-b704-73aa74c6417c-plugins-config-volume" pod "e482d2d9-f8db-4d89-b704-73aa74c6417c" (UID: "e482d2d9-f8db-4d89-b704-73aa74c6417c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:37.832939 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.832401    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-data" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.833981 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.833952    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-token" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:37.834501 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.834481    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-kube-api-access-n8psz" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "kube-api-access-n8psz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:37.834583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.834509    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e482d2d9-f8db-4d89-b704-73aa74c6417c-kube-api-access-zswm2" pod "e482d2d9-f8db-4d89-b704-73aa74c6417c" (UID: "e482d2d9-f8db-4d89-b704-73aa74c6417c"). InnerVolumeSpecName "kube-api-access-zswm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:37.834583 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.834509    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-envoy" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.835006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.834991    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-podinfo" pod "bbdd9f9d-6b83-495e-804f-e85d4185d396" (UID: "bbdd9f9d-6b83-495e-804f-e85d4185d396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933333    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e482d2d9-f8db-4d89-b704-73aa74c6417c-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933362    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8psz\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-kube-api-access-n8psz\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933372    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933382    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933390    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbdd9f9d-6b83-495e-804f-e85d4185d396-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933398    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zswm2\" (UniqueName: \"kubernetes.io/projected/e482d2d9-f8db-4d89-b704-73aa74c6417c-kube-api-access-zswm2\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933408    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933416    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.933417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:37.933424    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbdd9f9d-6b83-495e-804f-e85d4185d396-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:38.394187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.394094    4633 generic.go:356] "Generic (PLEG): container finished" podID="fef2c07d-1cb3-4b21-a25f-a525adb9a011" containerID="c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c" exitCode=0
Oct 30 21:56:38.394187 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.394150    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"
Oct 30 21:56:38.394440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.394180    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf" event={"ID":"fef2c07d-1cb3-4b21-a25f-a525adb9a011","Type":"ContainerDied","Data":"c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c"}
Oct 30 21:56:38.394440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.394233    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf" event={"ID":"fef2c07d-1cb3-4b21-a25f-a525adb9a011","Type":"ContainerDied","Data":"3df67936207ee7c896f606a643cf52a23f8954e9746284dfdb9afd603ad905fc"}
Oct 30 21:56:38.394440 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.394260    4633 scope.go:122] "RemoveContainer" containerID="c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c"
Oct 30 21:56:38.395868 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.395829    4633 generic.go:356] "Generic (PLEG): container finished" podID="e482d2d9-f8db-4d89-b704-73aa74c6417c" containerID="0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b" exitCode=0
Oct 30 21:56:38.396026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.395897    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"
Oct 30 21:56:38.396026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.395914    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq" event={"ID":"e482d2d9-f8db-4d89-b704-73aa74c6417c","Type":"ContainerDied","Data":"0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b"}
Oct 30 21:56:38.396026 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.395952    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq" event={"ID":"e482d2d9-f8db-4d89-b704-73aa74c6417c","Type":"ContainerDied","Data":"437336a92e7816196ab39dfba0fa31daf5a3064964a0b135876966f05ec4ec02"}
Oct 30 21:56:38.397785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.397760    4633 generic.go:356] "Generic (PLEG): container finished" podID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerID="0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164" exitCode=0
Oct 30 21:56:38.397926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.397811    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"
Oct 30 21:56:38.397926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.397843    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" event={"ID":"bbdd9f9d-6b83-495e-804f-e85d4185d396","Type":"ContainerDied","Data":"0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164"}
Oct 30 21:56:38.397926 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.397878    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq" event={"ID":"bbdd9f9d-6b83-495e-804f-e85d4185d396","Type":"ContainerDied","Data":"d24df9048c2e11645f203c5de68f2b363820eb6f2df1b791ea3ff7546c16ff0b"}
Oct 30 21:56:38.405823 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.405802    4633 scope.go:122] "RemoveContainer" containerID="c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c"
Oct 30 21:56:38.406100 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:38.406082    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c\": container with ID starting with c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c not found: ID does not exist" containerID="c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c"
Oct 30 21:56:38.406158 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.406108    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c"} err="failed to get container status \"c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c\": rpc error: code = NotFound desc = could not find container \"c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c\": container with ID starting with c98d0227cb32db9815fc571583f75fa9372fee8033b063691fe69fb1f6bb601c not found: ID does not exist"
Oct 30 21:56:38.406158 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.406125    4633 scope.go:122] "RemoveContainer" containerID="0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b"
Oct 30 21:56:38.425307 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.425281    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"]
Oct 30 21:56:38.427279 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.427258    4633 scope.go:122] "RemoveContainer" containerID="0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b"
Oct 30 21:56:38.427596 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:38.427575    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b\": container with ID starting with 0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b not found: ID does not exist" containerID="0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b"
Oct 30 21:56:38.427662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.427606    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b"} err="failed to get container status \"0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b\": rpc error: code = NotFound desc = could not find container \"0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b\": container with ID starting with 0b07e5b53ddab29a24e13a5ad2e7741e5ebd813b6e86ba53cdc5a896284af44b not found: ID does not exist"
Oct 30 21:56:38.427662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.427632    4633 scope.go:122] "RemoveContainer" containerID="0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164"
Oct 30 21:56:38.432589 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.432564    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3011c6-9449cd9bc-8nmtf"]
Oct 30 21:56:38.438200 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.438172    4633 scope.go:122] "RemoveContainer" containerID="0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164"
Oct 30 21:56:38.438515 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:38.438494    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164\": container with ID starting with 0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164 not found: ID does not exist" containerID="0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164"
Oct 30 21:56:38.438612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.438522    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164"} err="failed to get container status \"0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164\": rpc error: code = NotFound desc = could not find container \"0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164\": container with ID starting with 0b3f14da085c589996ae1b4975f8acf7ab1c36e37a7a4513be14316be15bf164 not found: ID does not exist"
Oct 30 21:56:38.454318 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.454280    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"]
Oct 30 21:56:38.457655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.457626    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3011c6-6769597779-g8wxq"]
Oct 30 21:56:38.472014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.471982    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"]
Oct 30 21:56:38.477723 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:38.477696    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-6g7mq"]
Oct 30 21:56:39.452477 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:39.452442    4633 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:56:39.452995 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:39.452505    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:56:39.790254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:39.790173    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" path="/var/lib/kubelet/pods/bbdd9f9d-6b83-495e-804f-e85d4185d396/volumes"
Oct 30 21:56:39.790564 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:39.790551    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e482d2d9-f8db-4d89-b704-73aa74c6417c" path="/var/lib/kubelet/pods/e482d2d9-f8db-4d89-b704-73aa74c6417c/volumes"
Oct 30 21:56:39.790905 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:39.790893    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef2c07d-1cb3-4b21-a25f-a525adb9a011" path="/var/lib/kubelet/pods/fef2c07d-1cb3-4b21-a25f-a525adb9a011/volumes"
Oct 30 21:56:40.749777 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.749755    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:56:40.865849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.865765    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-kube-api-access-q9dld\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-kube-api-access-q9dld\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.865849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.865814    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-token\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-token\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.865849 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.865839    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istiod-ca-cert\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.866144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.865863    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-certs\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.866144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.865884    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-data\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.866144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.865937    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-credential-socket\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.866144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.865979    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-envoy\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.866144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866048    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-socket\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.866144 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866093    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-podinfo\") pod \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\" (UID: \"ef74e2dc-1fb5-414d-b969-5edc2fb704ef\") "
Oct 30 21:56:40.866491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866241    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-data" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.866491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866278    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-credential-socket" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.866491 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866321    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istiod-ca-cert" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:40.866662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866496    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-socket" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.866662 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866520    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-certs" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.867021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866923    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.867021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866951    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.867021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866967    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.867021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866983    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.867021 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.866997    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.868482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.868460    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-kube-api-access-q9dld" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "kube-api-access-q9dld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:40.868482 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.868465    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-podinfo" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:40.868663 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.868489    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-token" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:40.868910 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.868891    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-envoy" pod "ef74e2dc-1fb5-414d-b969-5edc2fb704ef" (UID: "ef74e2dc-1fb5-414d-b969-5edc2fb704ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.968429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.968395    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9dld\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-kube-api-access-q9dld\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.968429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.968427    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.968429 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.968437    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.968704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:40.968448    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef74e2dc-1fb5-414d-b969-5edc2fb704ef-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:41.423648 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.423615    4633 generic.go:356] "Generic (PLEG): container finished" podID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerID="cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda" exitCode=0
Oct 30 21:56:41.423866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.423661    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" event={"ID":"ef74e2dc-1fb5-414d-b969-5edc2fb704ef","Type":"ContainerDied","Data":"cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda"}
Oct 30 21:56:41.423866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.423664    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"
Oct 30 21:56:41.423866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.423733    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw" event={"ID":"ef74e2dc-1fb5-414d-b969-5edc2fb704ef","Type":"ContainerDied","Data":"fe5dfc7e6d2d373d4b37abde98016ead6bb247019efb2d0c7cd53f27abbaab2d"}
Oct 30 21:56:41.423866 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.423764    4633 scope.go:122] "RemoveContainer" containerID="cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda"
Oct 30 21:56:41.439497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.439468    4633 scope.go:122] "RemoveContainer" containerID="cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda"
Oct 30 21:56:41.439844 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:41.439823    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda\": container with ID starting with cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda not found: ID does not exist" containerID="cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda"
Oct 30 21:56:41.439925 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.439851    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda"} err="failed to get container status \"cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda\": rpc error: code = NotFound desc = could not find container \"cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda\": container with ID starting with cd6e646d665f2f0eafe72b0bf1a1f857267ffae0b3e3b7aa98bf312f91c3ecda not found: ID does not exist"
Oct 30 21:56:41.453373 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.453350    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"]
Oct 30 21:56:41.457192 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.457170    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-z2qmw"]
Oct 30 21:56:41.791476 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.791383    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" path="/var/lib/kubelet/pods/ef74e2dc-1fb5-414d-b969-5edc2fb704ef/volumes"
Oct 30 21:56:41.795081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:41.795053    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:56:47.365735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.365698    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"]
Oct 30 21:56:47.366298 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.365982    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx" podUID="8d36d0a1-89c6-4aa8-8e38-0d18da409c93" containerName="maas-api" containerID="cri-o://b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762" gracePeriod=30
Oct 30 21:56:47.622772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.622699    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:56:47.765321 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.765290    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-kube-api-access-rvpr6\" (UniqueName: \"kubernetes.io/projected/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-kube-api-access-rvpr6\") pod \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") "
Oct 30 21:56:47.765506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.765382    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-metrics-certs\") pod \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") "
Oct 30 21:56:47.765506 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.765443    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-openshift-service-ca\") pod \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") "
Oct 30 21:56:47.765709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.765544    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-maas-api-tls\") pod \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\" (UID: \"8d36d0a1-89c6-4aa8-8e38-0d18da409c93\") "
Oct 30 21:56:47.766209 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.766127    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-openshift-service-ca" pod "8d36d0a1-89c6-4aa8-8e38-0d18da409c93" (UID: "8d36d0a1-89c6-4aa8-8e38-0d18da409c93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:47.767941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.767909    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-kube-api-access-rvpr6" pod "8d36d0a1-89c6-4aa8-8e38-0d18da409c93" (UID: "8d36d0a1-89c6-4aa8-8e38-0d18da409c93"). InnerVolumeSpecName "kube-api-access-rvpr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:47.768322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.768299    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-metrics-certs" pod "8d36d0a1-89c6-4aa8-8e38-0d18da409c93" (UID: "8d36d0a1-89c6-4aa8-8e38-0d18da409c93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:47.768400 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.768325    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-maas-api-tls" pod "8d36d0a1-89c6-4aa8-8e38-0d18da409c93" (UID: "8d36d0a1-89c6-4aa8-8e38-0d18da409c93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:47.867621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.867579    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvpr6\" (UniqueName: \"kubernetes.io/projected/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-kube-api-access-rvpr6\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:47.867621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.867611    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:47.867621 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.867625    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:47.867936 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:47.867639    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d36d0a1-89c6-4aa8-8e38-0d18da409c93-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:48.476287 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.476252    4633 generic.go:356] "Generic (PLEG): container finished" podID="8d36d0a1-89c6-4aa8-8e38-0d18da409c93" containerID="b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762" exitCode=0
Oct 30 21:56:48.476837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.476298    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"
Oct 30 21:56:48.476837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.476327    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx" event={"ID":"8d36d0a1-89c6-4aa8-8e38-0d18da409c93","Type":"ContainerDied","Data":"b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762"}
Oct 30 21:56:48.476837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.476383    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx" event={"ID":"8d36d0a1-89c6-4aa8-8e38-0d18da409c93","Type":"ContainerDied","Data":"6dc831cbbbc84440e00a43ba77a5e98a98b2915763cb708bf359fd07992e37bb"}
Oct 30 21:56:48.476837 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.476404    4633 scope.go:122] "RemoveContainer" containerID="b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762"
Oct 30 21:56:48.487459 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.487434    4633 scope.go:122] "RemoveContainer" containerID="b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762"
Oct 30 21:56:48.487764 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:48.487736    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762\": container with ID starting with b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762 not found: ID does not exist" containerID="b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762"
Oct 30 21:56:48.487830 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.487774    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762"} err="failed to get container status \"b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762\": rpc error: code = NotFound desc = could not find container \"b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762\": container with ID starting with b61d65c6a683f4196459951dc42fdc7648e6685d3065167c604f5e8ac83ba762 not found: ID does not exist"
Oct 30 21:56:48.507485 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.507443    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"]
Oct 30 21:56:48.517958 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:48.517932    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0ea1ea-85cbdf8994-s8dcx"]
Oct 30 21:56:49.790115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:49.790073    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d36d0a1-89c6-4aa8-8e38-0d18da409c93" path="/var/lib/kubelet/pods/8d36d0a1-89c6-4aa8-8e38-0d18da409c93/volumes"
Oct 30 21:56:55.015050 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:56:55.015009    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:56:59.279112 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:56:59.279076    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:57:00.540208 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:00.540168    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"]
Oct 30 21:57:00.540765 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:00.540387    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy" containerID="cri-o://a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893" gracePeriod=30
Oct 30 21:57:00.672261 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:00.672217    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"]
Oct 30 21:57:00.672509 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:00.672483    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy" containerID="cri-o://bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0" gracePeriod=30
Oct 30 21:57:05.847519 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.847485    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:57:05.884426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884360    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-socket\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.884426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884423    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-token\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-token\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.884655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884468    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-kube-api-access-b2h9j\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.884655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884506    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-credential-socket\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.884655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884594    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-envoy\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.884655 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884637    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ef3b75-43f3-44ea-a266-7184d31c4331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ef3b75-43f3-44ea-a266-7184d31c4331-istiod-ca-cert\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.885045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884697    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-podinfo\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.885045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884730    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-certs\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.885045 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.884804    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-data\") pod \"54ef3b75-43f3-44ea-a266-7184d31c4331\" (UID: \"54ef3b75-43f3-44ea-a266-7184d31c4331\") "
Oct 30 21:57:05.885575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.885356    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-credential-socket" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.885575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.885477    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-certs" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.885575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.885497    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-data" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.887232 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.887208    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ef3b75-43f3-44ea-a266-7184d31c4331-istiod-ca-cert" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:05.887873 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.887846    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-socket" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.889042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.888997    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-kube-api-access-b2h9j" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "kube-api-access-b2h9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:05.891624 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.891590    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-podinfo" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:05.892497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.892471    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-envoy" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.894528 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.894501    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-token" pod "54ef3b75-43f3-44ea-a266-7184d31c4331" (UID: "54ef3b75-43f3-44ea-a266-7184d31c4331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:05.953457 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.953428    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:57:05.986741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.986702    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-istio-token\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-istio-token\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.986935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.986768    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-envoy\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.986935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.986839    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-socket\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.986935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.986868    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ddf97db-429a-47db-b503-56559682eb9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ddf97db-429a-47db-b503-56559682eb9c-istiod-ca-cert\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.986935 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.986930    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-credential-socket\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.987337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.986971    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-kube-api-access-6k7mw\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-kube-api-access-6k7mw\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.987337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987018    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ddf97db-429a-47db-b503-56559682eb9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ddf97db-429a-47db-b503-56559682eb9c-istio-podinfo\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.987337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987075    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-certs\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.987337 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987159    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-data\") pod \"9ddf97db-429a-47db-b503-56559682eb9c\" (UID: \"9ddf97db-429a-47db-b503-56559682eb9c\") "
Oct 30 21:57:05.987609 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987589    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.987653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987616    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ef3b75-43f3-44ea-a266-7184d31c4331-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.987653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987633    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.987653 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987646    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.987822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987661    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.987822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987708    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.987822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987722    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.987822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987737    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2h9j\" (UniqueName: \"kubernetes.io/projected/54ef3b75-43f3-44ea-a266-7184d31c4331-kube-api-access-b2h9j\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.987822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.987752    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ef3b75-43f3-44ea-a266-7184d31c4331-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.988039 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.988015    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-data" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.988763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.988734    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-credential-socket" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.989442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.989277    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-certs" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.989811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.989594    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-socket" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.990115 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.990056    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ddf97db-429a-47db-b503-56559682eb9c-istiod-ca-cert" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:05.991092 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.991065    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-envoy" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.991207 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.991143    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-istio-token" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:05.991811 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.991786    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ddf97db-429a-47db-b503-56559682eb9c-istio-podinfo" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:05.992937 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:05.992912    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-kube-api-access-6k7mw" pod "9ddf97db-429a-47db-b503-56559682eb9c" (UID: "9ddf97db-429a-47db-b503-56559682eb9c"). InnerVolumeSpecName "kube-api-access-6k7mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:06.088998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.088891    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ddf97db-429a-47db-b503-56559682eb9c-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.088998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.088931    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.088998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.088946    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.088998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.088960    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.088998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.088974    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.088998 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.088990    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.089497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.089004    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ddf97db-429a-47db-b503-56559682eb9c-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.089497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.089019    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ddf97db-429a-47db-b503-56559682eb9c-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.089497 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.089032    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k7mw\" (UniqueName: \"kubernetes.io/projected/9ddf97db-429a-47db-b503-56559682eb9c-kube-api-access-6k7mw\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.618641 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.618608    4633 generic.go:356] "Generic (PLEG): container finished" podID="9ddf97db-429a-47db-b503-56559682eb9c" containerID="bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0" exitCode=0
Oct 30 21:57:06.618861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.618661    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"
Oct 30 21:57:06.618861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.618705    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" event={"ID":"9ddf97db-429a-47db-b503-56559682eb9c","Type":"ContainerDied","Data":"bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0"}
Oct 30 21:57:06.618861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.618755    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4" event={"ID":"9ddf97db-429a-47db-b503-56559682eb9c","Type":"ContainerDied","Data":"4ca4ec6558509e8f2aecf3b9b650cb2f4e0f09cef149bfce99067f7db81bc2a3"}
Oct 30 21:57:06.618861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.618776    4633 scope.go:122] "RemoveContainer" containerID="bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0"
Oct 30 21:57:06.620853 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.620830    4633 generic.go:356] "Generic (PLEG): container finished" podID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerID="a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893" exitCode=0
Oct 30 21:57:06.621009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.620873    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"
Oct 30 21:57:06.621009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.620889    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" event={"ID":"54ef3b75-43f3-44ea-a266-7184d31c4331","Type":"ContainerDied","Data":"a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893"}
Oct 30 21:57:06.621009 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.620926    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4" event={"ID":"54ef3b75-43f3-44ea-a266-7184d31c4331","Type":"ContainerDied","Data":"d140c3012044ba804d4f65909cf9a0769a6cf04aac109e0c456c51dbad2263b4"}
Oct 30 21:57:06.631510 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.631488    4633 scope.go:122] "RemoveContainer" containerID="bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0"
Oct 30 21:57:06.631922 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:57:06.631900    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0\": container with ID starting with bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0 not found: ID does not exist" containerID="bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0"
Oct 30 21:57:06.632006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.631938    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0"} err="failed to get container status \"bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0\": rpc error: code = NotFound desc = could not find container \"bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0\": container with ID starting with bcb19da22a15df1e307b060b13c1035e5c6ff7bb5055cc0eacf389ba722672d0 not found: ID does not exist"
Oct 30 21:57:06.632006 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.631954    4633 scope.go:122] "RemoveContainer" containerID="a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893"
Oct 30 21:57:06.641116 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.641098    4633 scope.go:122] "RemoveContainer" containerID="cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649"
Oct 30 21:57:06.647137 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.647117    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"]
Oct 30 21:57:06.650996 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.650975    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-q9zt4"]
Oct 30 21:57:06.651592 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.651575    4633 scope.go:122] "RemoveContainer" containerID="a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893"
Oct 30 21:57:06.651929 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:57:06.651912    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893\": container with ID starting with a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893 not found: ID does not exist" containerID="a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893"
Oct 30 21:57:06.651968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.651937    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893"} err="failed to get container status \"a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893\": rpc error: code = NotFound desc = could not find container \"a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893\": container with ID starting with a5cb596700c5bf6bfd6dd81e58123d7a567e1b8e8d487c9de9a0624cd9dc2893 not found: ID does not exist"
Oct 30 21:57:06.651968 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.651956    4633 scope.go:122] "RemoveContainer" containerID="cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649"
Oct 30 21:57:06.652212 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:57:06.652191    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649\": container with ID starting with cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649 not found: ID does not exist" containerID="cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649"
Oct 30 21:57:06.652256 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.652219    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649"} err="failed to get container status \"cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649\": rpc error: code = NotFound desc = could not find container \"cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649\": container with ID starting with cf8ebfd7cea185d55d6e5bc4cf578712601ee1aba3f901c2036cc03b01085649 not found: ID does not exist"
Oct 30 21:57:06.665329 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.665305    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"]
Oct 30 21:57:06.673027 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:06.673000    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-s8xq4"]
Oct 30 21:57:07.790110 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:07.790070    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" path="/var/lib/kubelet/pods/54ef3b75-43f3-44ea-a266-7184d31c4331/volumes"
Oct 30 21:57:07.790616 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:07.790557    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ddf97db-429a-47db-b503-56559682eb9c" path="/var/lib/kubelet/pods/9ddf97db-429a-47db-b503-56559682eb9c/volumes"
Oct 30 21:57:11.113553 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:11.113514    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"]
Oct 30 21:57:14.123082 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.123048    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"]
Oct 30 21:57:14.123572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.123301    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn" podUID="749359a0-c74c-4899-8505-89f60d2dc987" containerName="maas-api" containerID="cri-o://95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e" gracePeriod=30
Oct 30 21:57:14.446763 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.446732    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:57:14.488148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.488112    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-metrics-certs\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-metrics-certs\") pod \"749359a0-c74c-4899-8505-89f60d2dc987\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") "
Oct 30 21:57:14.488361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.488204    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-maas-api-tls\") pod \"749359a0-c74c-4899-8505-89f60d2dc987\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") "
Oct 30 21:57:14.488361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.488308    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749359a0-c74c-4899-8505-89f60d2dc987-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/749359a0-c74c-4899-8505-89f60d2dc987-openshift-service-ca\") pod \"749359a0-c74c-4899-8505-89f60d2dc987\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") "
Oct 30 21:57:14.488494 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.488375    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749359a0-c74c-4899-8505-89f60d2dc987-kube-api-access-f7mrb\" (UniqueName: \"kubernetes.io/projected/749359a0-c74c-4899-8505-89f60d2dc987-kube-api-access-f7mrb\") pod \"749359a0-c74c-4899-8505-89f60d2dc987\" (UID: \"749359a0-c74c-4899-8505-89f60d2dc987\") "
Oct 30 21:57:14.488812 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.488780    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749359a0-c74c-4899-8505-89f60d2dc987-openshift-service-ca" pod "749359a0-c74c-4899-8505-89f60d2dc987" (UID: "749359a0-c74c-4899-8505-89f60d2dc987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:14.490481 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.490453    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-metrics-certs" pod "749359a0-c74c-4899-8505-89f60d2dc987" (UID: "749359a0-c74c-4899-8505-89f60d2dc987"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:14.490779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.490652    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-maas-api-tls" pod "749359a0-c74c-4899-8505-89f60d2dc987" (UID: "749359a0-c74c-4899-8505-89f60d2dc987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:14.490779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.490655    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749359a0-c74c-4899-8505-89f60d2dc987-kube-api-access-f7mrb" pod "749359a0-c74c-4899-8505-89f60d2dc987" (UID: "749359a0-c74c-4899-8505-89f60d2dc987"). InnerVolumeSpecName "kube-api-access-f7mrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:14.590246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.590214    4633 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/749359a0-c74c-4899-8505-89f60d2dc987-openshift-service-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:14.590246 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.590242    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7mrb\" (UniqueName: \"kubernetes.io/projected/749359a0-c74c-4899-8505-89f60d2dc987-kube-api-access-f7mrb\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:14.590455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.590255    4633 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-metrics-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:14.590455 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.590266    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/749359a0-c74c-4899-8505-89f60d2dc987-maas-api-tls\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:14.678752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.678635    4633 generic.go:356] "Generic (PLEG): container finished" podID="749359a0-c74c-4899-8505-89f60d2dc987" containerID="95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e" exitCode=0
Oct 30 21:57:14.678752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.678698    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"
Oct 30 21:57:14.678752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.678726    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn" event={"ID":"749359a0-c74c-4899-8505-89f60d2dc987","Type":"ContainerDied","Data":"95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e"}
Oct 30 21:57:14.679030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.678784    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn" event={"ID":"749359a0-c74c-4899-8505-89f60d2dc987","Type":"ContainerDied","Data":"e08e201952d27f88495095966a8c8c10d57da949101355dd5433e7595ca18751"}
Oct 30 21:57:14.679030 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.678808    4633 scope.go:122] "RemoveContainer" containerID="95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e"
Oct 30 21:57:14.690751 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.690726    4633 scope.go:122] "RemoveContainer" containerID="95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e"
Oct 30 21:57:14.691069 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:57:14.691018    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e\": container with ID starting with 95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e not found: ID does not exist" containerID="95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e"
Oct 30 21:57:14.691069 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.691050    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e"} err="failed to get container status \"95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e\": rpc error: code = NotFound desc = could not find container \"95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e\": container with ID starting with 95c0a7f82709669729885957b6ef8bba628e9f4c97a5cd8a03435ab313c9172e not found: ID does not exist"
Oct 30 21:57:14.705547 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.705519    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"]
Oct 30 21:57:14.710306 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:14.710284    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-06bb0f-77fb8c4fb4-w58rn"]
Oct 30 21:57:15.791210 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:15.791172    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749359a0-c74c-4899-8505-89f60d2dc987" path="/var/lib/kubelet/pods/749359a0-c74c-4899-8505-89f60d2dc987/volumes"
Oct 30 21:57:16.115014 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:16.114916    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b" podUID="57b7ef66-3f64-4331-88cf-6825ab28d5dc" containerName="payload-processing" containerID="cri-o://e2321fa8ee454b8f94bcdb6e81a15716dc0e464074af62b1a4380fbcafd9ec0e" gracePeriod=25
Oct 30 21:57:16.681923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:16.681875    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b" podUID="57b7ef66-3f64-4331-88cf-6825ab28d5dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:17.156008 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:17.155967    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:57:22.383335 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:22.383299    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"]
Oct 30 21:57:22.383933 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:22.383541    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy" containerID="cri-o://87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968" gracePeriod=30
Oct 30 21:57:25.014844 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:57:25.014797    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:57:26.681900 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:26.681861    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b" podUID="57b7ef66-3f64-4331-88cf-6825ab28d5dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:27.652575 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.652547    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:57:27.729186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729103    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-podinfo\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729186 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729147    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istiod-ca-cert\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729194    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-envoy\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729245    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-credential-socket\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729369    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-socket\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729453    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-data\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729497    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-certs\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729551    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-token\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-token\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729589    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-credential-socket" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729597    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-kube-api-access-qjl5w\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-kube-api-access-qjl5w\") pod \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\" (UID: \"62d132a9-f682-4d56-9de2-aaaa58c7eb35\") "
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729641    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-socket" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.729804 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.729659    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istiod-ca-cert" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:27.730282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.730063    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-certs" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.730282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.730086    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.730282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.730108    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.730282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.730125    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.730282 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.730200    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-data" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.731540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.731509    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-podinfo" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:27.731540 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.731522    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-envoy" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.732228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.732204    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-kube-api-access-qjl5w" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "kube-api-access-qjl5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:27.732228 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.732214    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-token" pod "62d132a9-f682-4d56-9de2-aaaa58c7eb35" (UID: "62d132a9-f682-4d56-9de2-aaaa58c7eb35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:27.774441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.774406    4633 generic.go:356] "Generic (PLEG): container finished" podID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerID="87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968" exitCode=0
Oct 30 21:57:27.774629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.774464    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"
Oct 30 21:57:27.774629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.774487    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" event={"ID":"62d132a9-f682-4d56-9de2-aaaa58c7eb35","Type":"ContainerDied","Data":"87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968"}
Oct 30 21:57:27.774629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.774542    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9" event={"ID":"62d132a9-f682-4d56-9de2-aaaa58c7eb35","Type":"ContainerDied","Data":"68fccfef0d299348c9e7a3d4e1b3923284bc043c77b846f4f3c1d5281674508b"}
Oct 30 21:57:27.774629 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.774565    4633 scope.go:122] "RemoveContainer" containerID="87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968"
Oct 30 21:57:27.786518 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.786501    4633 scope.go:122] "RemoveContainer" containerID="b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1"
Oct 30 21:57:27.796808 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.796786    4633 scope.go:122] "RemoveContainer" containerID="87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968"
Oct 30 21:57:27.797107 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:57:27.797085    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968\": container with ID starting with 87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968 not found: ID does not exist" containerID="87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968"
Oct 30 21:57:27.797168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.797116    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968"} err="failed to get container status \"87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968\": rpc error: code = NotFound desc = could not find container \"87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968\": container with ID starting with 87bdcf4531a5dc40324472ede1fc9fe33f1d7101a828d821058d6a049494d968 not found: ID does not exist"
Oct 30 21:57:27.797168 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.797135    4633 scope.go:122] "RemoveContainer" containerID="b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1"
Oct 30 21:57:27.797361 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:57:27.797343    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1\": container with ID starting with b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1 not found: ID does not exist" containerID="b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1"
Oct 30 21:57:27.797406 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.797365    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1"} err="failed to get container status \"b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1\": rpc error: code = NotFound desc = could not find container \"b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1\": container with ID starting with b1bf07695e0519e3e62c109811fa307bc24f1070695e9e85068a229883ff01f1 not found: ID does not exist"
Oct 30 21:57:27.798794 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.798775    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"]
Oct 30 21:57:27.807612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.807588    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-b46x9"]
Oct 30 21:57:27.831612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.831582    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.831612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.831613    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.831612 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.831629    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.831959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.831641    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjl5w\" (UniqueName: \"kubernetes.io/projected/62d132a9-f682-4d56-9de2-aaaa58c7eb35-kube-api-access-qjl5w\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.831959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.831657    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.831959 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:27.831696    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62d132a9-f682-4d56-9de2-aaaa58c7eb35-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:29.789384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:29.789348    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" path="/var/lib/kubelet/pods/62d132a9-f682-4d56-9de2-aaaa58c7eb35/volumes"
Oct 30 21:57:36.361037 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.361012    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:57:36.519117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.519085    4633 scope.go:122] "RemoveContainer" containerID="e2321fa8ee454b8f94bcdb6e81a15716dc0e464074af62b1a4380fbcafd9ec0e"
Oct 30 21:57:36.526861 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.526835    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b7ef66-3f64-4331-88cf-6825ab28d5dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b7ef66-3f64-4331-88cf-6825ab28d5dc-plugins-config-volume\") pod \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\" (UID: \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\") "
Oct 30 21:57:36.527036 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.527020    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b7ef66-3f64-4331-88cf-6825ab28d5dc-kube-api-access-gv564\" (UniqueName: \"kubernetes.io/projected/57b7ef66-3f64-4331-88cf-6825ab28d5dc-kube-api-access-gv564\") pod \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\" (UID: \"57b7ef66-3f64-4331-88cf-6825ab28d5dc\") "
Oct 30 21:57:36.527233 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.527210    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b7ef66-3f64-4331-88cf-6825ab28d5dc-plugins-config-volume" pod "57b7ef66-3f64-4331-88cf-6825ab28d5dc" (UID: "57b7ef66-3f64-4331-88cf-6825ab28d5dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:36.527428 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.527406    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57b7ef66-3f64-4331-88cf-6825ab28d5dc-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:36.529715 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.529664    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b7ef66-3f64-4331-88cf-6825ab28d5dc-kube-api-access-gv564" pod "57b7ef66-3f64-4331-88cf-6825ab28d5dc" (UID: "57b7ef66-3f64-4331-88cf-6825ab28d5dc"). InnerVolumeSpecName "kube-api-access-gv564". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:36.628800 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.628768    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv564\" (UniqueName: \"kubernetes.io/projected/57b7ef66-3f64-4331-88cf-6825ab28d5dc-kube-api-access-gv564\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:36.830248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.830166    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"
Oct 30 21:57:36.830248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.830185    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b" event={"ID":"57b7ef66-3f64-4331-88cf-6825ab28d5dc","Type":"ContainerDied","Data":"e2321fa8ee454b8f94bcdb6e81a15716dc0e464074af62b1a4380fbcafd9ec0e"}
Oct 30 21:57:36.830248 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.830236    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b" event={"ID":"57b7ef66-3f64-4331-88cf-6825ab28d5dc","Type":"ContainerDied","Data":"b8af609d043b96bbea1983c07c93a84f7f2e71847894e81b7ea85ba444c3e921"}
Oct 30 21:57:36.874106 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.874075    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"]
Oct 30 21:57:36.878433 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:36.878399    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-06bb0f-6f546f44b5-w2m4b"]
Oct 30 21:57:37.726775 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:37.726738    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:57:37.803903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:37.803660    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b7ef66-3f64-4331-88cf-6825ab28d5dc" path="/var/lib/kubelet/pods/57b7ef66-3f64-4331-88cf-6825ab28d5dc/volumes"
Oct 30 21:57:43.667844 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:43.667743    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"]
Oct 30 21:57:43.668515 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:43.668037    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy" containerID="cri-o://1fdc524fc15d2c5f27cb864fe49f773bee01b24fb57b6ebd9f552af03bd88c5f" gracePeriod=30
Oct 30 21:57:48.909987 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.909952    4633 generic.go:356] "Generic (PLEG): container finished" podID="2ae502d4-f440-470b-a8f9-957c672d6745" containerID="1fdc524fc15d2c5f27cb864fe49f773bee01b24fb57b6ebd9f552af03bd88c5f" exitCode=0
Oct 30 21:57:48.910382 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.910028    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" event={"ID":"2ae502d4-f440-470b-a8f9-957c672d6745","Type":"ContainerDied","Data":"1fdc524fc15d2c5f27cb864fe49f773bee01b24fb57b6ebd9f552af03bd88c5f"}
Oct 30 21:57:48.936443 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.936415    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.970874    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae502d4-f440-470b-a8f9-957c672d6745-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae502d4-f440-470b-a8f9-957c672d6745-istiod-ca-cert\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.970924    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ae502d4-f440-470b-a8f9-957c672d6745-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae502d4-f440-470b-a8f9-957c672d6745-istio-podinfo\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.971038    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-credential-socket\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.971062    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-envoy\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.971119    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-istio-token\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-istio-token\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.971143    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-data\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.971160    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-certs\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.971188    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-kube-api-access-2f4fh\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-kube-api-access-2f4fh\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.971211    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-socket\") pod \"2ae502d4-f440-470b-a8f9-957c672d6745\" (UID: \"2ae502d4-f440-470b-a8f9-957c672d6745\") "
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.971729    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-socket" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:48.972145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.972094    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-certs" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:48.973081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.972246    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-data" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:48.973081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.972453    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae502d4-f440-470b-a8f9-957c672d6745-istiod-ca-cert" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:48.973081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.972509    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-credential-socket" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:48.974781 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.974750    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-envoy" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:48.975884 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.975858    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-kube-api-access-2f4fh" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "kube-api-access-2f4fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:48.976011 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.975988    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-istio-token" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:48.976264 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:48.976245    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae502d4-f440-470b-a8f9-957c672d6745-istio-podinfo" pod "2ae502d4-f440-470b-a8f9-957c672d6745" (UID: "2ae502d4-f440-470b-a8f9-957c672d6745"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:49.073218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073186    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.073218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073216    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.073218 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073227    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f4fh\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-kube-api-access-2f4fh\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.073493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073236    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.073493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073245    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae502d4-f440-470b-a8f9-957c672d6745-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.073493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073253    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae502d4-f440-470b-a8f9-957c672d6745-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.073493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073262    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.073493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073270    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae502d4-f440-470b-a8f9-957c672d6745-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.073493 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.073280    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae502d4-f440-470b-a8f9-957c672d6745-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.182927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.182885    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"]
Oct 30 21:57:49.917646 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.917611    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"
Oct 30 21:57:49.918150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.917655    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6" event={"ID":"2ae502d4-f440-470b-a8f9-957c672d6745","Type":"ContainerDied","Data":"f65f1005b0dc0c13ab6d547e22d2a1408731d5d81888b8233054f2a687074fa2"}
Oct 30 21:57:49.918150 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.917747    4633 scope.go:122] "RemoveContainer" containerID="1fdc524fc15d2c5f27cb864fe49f773bee01b24fb57b6ebd9f552af03bd88c5f"
Oct 30 21:57:49.936994 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.936968    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"]
Oct 30 21:57:49.940783 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:49.940761    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-zshs6"]
Oct 30 21:57:51.790089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:51.790054    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" path="/var/lib/kubelet/pods/2ae502d4-f440-470b-a8f9-957c672d6745/volumes"
Oct 30 21:57:54.556741 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:54.556702    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"]
Oct 30 21:57:54.566316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:54.566290    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"]
Oct 30 21:57:55.014641 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:57:55.014596    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:57:55.239314 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239277    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"]
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239892    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f8023d07-bf60-418f-b26c-1ef210b5f032" containerName="main"
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239917    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8023d07-bf60-418f-b26c-1ef210b5f032" containerName="main"
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239935    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e482d2d9-f8db-4d89-b704-73aa74c6417c" containerName="payload-pre-processing"
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239946    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e482d2d9-f8db-4d89-b704-73aa74c6417c" containerName="payload-pre-processing"
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239962    4633 cpu_manager.go:414] "containerMap: removing container" podUID="fef2c07d-1cb3-4b21-a25f-a525adb9a011" containerName="payload-processing"
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239969    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef2c07d-1cb3-4b21-a25f-a525adb9a011" containerName="payload-processing"
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239978    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy"
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239987    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy"
Oct 30 21:57:55.239986 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.239999    4633 cpu_manager.go:414] "containerMap: removing container" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240004    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240016    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240021    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240028    4633 cpu_manager.go:414] "containerMap: removing container" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240033    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240040    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240044    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240049    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d5d863f5-b05e-4922-9f70-015323992554" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240055    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d863f5-b05e-4922-9f70-015323992554" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240061    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240065    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240070    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240075    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240083    4633 cpu_manager.go:414] "containerMap: removing container" podUID="0a3ac7f8-6268-40e8-9599-5c11cca9390d" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240087    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3ac7f8-6268-40e8-9599-5c11cca9390d" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240092    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ba5084b7-93d3-4851-92fd-af91e4ed7852" containerName="main"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240096    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5084b7-93d3-4851-92fd-af91e4ed7852" containerName="main"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240102    4633 cpu_manager.go:414] "containerMap: removing container" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240107    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240115    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240119    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240126    4633 cpu_manager.go:414] "containerMap: removing container" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240129    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240135    4633 cpu_manager.go:414] "containerMap: removing container" podUID="57b7ef66-3f64-4331-88cf-6825ab28d5dc" containerName="payload-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240140    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b7ef66-3f64-4331-88cf-6825ab28d5dc" containerName="payload-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240148    4633 cpu_manager.go:414] "containerMap: removing container" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240152    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240159    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c8f26023-41db-4a13-b6be-912de216ccc5" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240164    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f26023-41db-4a13-b6be-912de216ccc5" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240170    4633 cpu_manager.go:414] "containerMap: removing container" podUID="30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240175    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240182    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240187    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240194    4633 cpu_manager.go:414] "containerMap: removing container" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240198    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240208    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f4238ff6-1550-434d-979b-f44b9bd50a99" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240212    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4238ff6-1550-434d-979b-f44b9bd50a99" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240218    4633 cpu_manager.go:414] "containerMap: removing container" podUID="8d36d0a1-89c6-4aa8-8e38-0d18da409c93" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240223    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d36d0a1-89c6-4aa8-8e38-0d18da409c93" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240230    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e42295e3-7338-4654-ae73-c08cfa9a4875" containerName="payload-pre-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240234    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42295e3-7338-4654-ae73-c08cfa9a4875" containerName="payload-pre-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240242    4633 cpu_manager.go:414] "containerMap: removing container" podUID="95f4f687-e025-46fd-85a5-9a93cfef9cc6" containerName="payload-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240247    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f4f687-e025-46fd-85a5-9a93cfef9cc6" containerName="payload-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240255    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240259    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240269    4633 cpu_manager.go:414] "containerMap: removing container" podUID="49fddf86-5424-4a8a-b896-ca1aafc24165" containerName="payload-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240276    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49fddf86-5424-4a8a-b896-ca1aafc24165" containerName="payload-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240286    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240291    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240300    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d4fe890f-059c-478e-a61a-8f8dc565b237" containerName="payload-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240306    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fe890f-059c-478e-a61a-8f8dc565b237" containerName="payload-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240316    4633 cpu_manager.go:414] "containerMap: removing container" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240322    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240330    4633 cpu_manager.go:414] "containerMap: removing container" podUID="5b8f84c7-6449-4add-be75-f2e3819e04a7" containerName="payload-pre-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240336    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8f84c7-6449-4add-be75-f2e3819e04a7" containerName="payload-pre-processing"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240346    4633 cpu_manager.go:414] "containerMap: removing container" podUID="af90c02b-8a15-411b-bbcb-d3d658effbec" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240353    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af90c02b-8a15-411b-bbcb-d3d658effbec" containerName="maas-api"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240362    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240369    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240378    4633 cpu_manager.go:414] "containerMap: removing container" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240385    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240393    4633 cpu_manager.go:414] "containerMap: removing container" podUID="7b0d20f7-3a1a-4875-84fb-29ae05de3304" containerName="main"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240400    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0d20f7-3a1a-4875-84fb-29ae05de3304" containerName="main"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240413    4633 cpu_manager.go:414] "containerMap: removing container" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240419    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240428    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240435    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240444    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240451    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240463    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240469    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240479    4633 cpu_manager.go:414] "containerMap: removing container" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240485    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240495    4633 cpu_manager.go:414] "containerMap: removing container" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy"
Oct 30 21:57:55.240467 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240501    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240512    4633 cpu_manager.go:414] "containerMap: removing container" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240519    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240530    4633 cpu_manager.go:414] "containerMap: removing container" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240536    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240544    4633 cpu_manager.go:414] "containerMap: removing container" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240550    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240563    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240569    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240577    4633 cpu_manager.go:414] "containerMap: removing container" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240584    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240595    4633 cpu_manager.go:414] "containerMap: removing container" podUID="656dbefb-91f4-48ce-afe3-742b52686788" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240601    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656dbefb-91f4-48ce-afe3-742b52686788" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240610    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ba0f6281-84e4-4b7b-a7ec-2412bfb4da99" containerName="payload-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240616    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0f6281-84e4-4b7b-a7ec-2412bfb4da99" containerName="payload-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240625    4633 cpu_manager.go:414] "containerMap: removing container" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240632    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240643    4633 cpu_manager.go:414] "containerMap: removing container" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240649    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240658    4633 cpu_manager.go:414] "containerMap: removing container" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240664    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240692    4633 cpu_manager.go:414] "containerMap: removing container" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240698    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240711    4633 cpu_manager.go:414] "containerMap: removing container" podUID="475d261a-7603-40c1-8e1a-6e62ee35a5b9" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240718    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475d261a-7603-40c1-8e1a-6e62ee35a5b9" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240732    4633 cpu_manager.go:414] "containerMap: removing container" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240739    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240748    4633 cpu_manager.go:414] "containerMap: removing container" podUID="b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" containerName="main"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240753    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" containerName="main"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240762    4633 cpu_manager.go:414] "containerMap: removing container" podUID="9645737c-61cc-475d-9832-d5dda1b7dee9" containerName="main"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240767    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9645737c-61cc-475d-9832-d5dda1b7dee9" containerName="main"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240777    4633 cpu_manager.go:414] "containerMap: removing container" podUID="749359a0-c74c-4899-8505-89f60d2dc987" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240784    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749359a0-c74c-4899-8505-89f60d2dc987" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240793    4633 cpu_manager.go:414] "containerMap: removing container" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240799    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240807    4633 cpu_manager.go:414] "containerMap: removing container" podUID="676be30a-0353-4bbf-8775-1219db088582" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240814    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676be30a-0353-4bbf-8775-1219db088582" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240823    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240829    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240842    4633 cpu_manager.go:414] "containerMap: removing container" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240849    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240966    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e482d2d9-f8db-4d89-b704-73aa74c6417c" containerName="payload-pre-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240987    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef2c07d-1cb3-4b21-a25f-a525adb9a011" containerName="payload-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.240997    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241006    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="90cf3712-d04c-4816-9a32-3de435a3f6f7" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241015    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241023    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="df40741b-59ed-4b6d-95df-8b515a02eb61" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241034    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241042    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d863f5-b05e-4922-9f70-015323992554" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241053    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241063    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6592f635-0310-4fae-ad00-dec841dec9c7" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241070    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3ac7f8-6268-40e8-9599-5c11cca9390d" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241077    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5084b7-93d3-4851-92fd-af91e4ed7852" containerName="main"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241085    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241094    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241101    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241109    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b7ef66-3f64-4331-88cf-6825ab28d5dc" containerName="payload-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241117    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="244ce12a-d4e5-4ee7-9b57-14b1f13f345b" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241126    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f26023-41db-4a13-b6be-912de216ccc5" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241132    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d0a91f-367d-4fe9-9f3d-e6f1ed0a543e" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241139    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a521156d-a012-4fe7-8873-a30d24fa3bdd" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241145    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="510937e8-5b5e-474d-9d5b-452eb5062cb8" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241152    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4238ff6-1550-434d-979b-f44b9bd50a99" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241160    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d36d0a1-89c6-4aa8-8e38-0d18da409c93" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241168    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42295e3-7338-4654-ae73-c08cfa9a4875" containerName="payload-pre-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241176    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f4f687-e025-46fd-85a5-9a93cfef9cc6" containerName="payload-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241186    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c630ce-a087-471a-8916-1d97307638fd" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241197    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="49fddf86-5424-4a8a-b896-ca1aafc24165" containerName="payload-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241207    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241215    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fe890f-059c-478e-a61a-8f8dc565b237" containerName="payload-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241225    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241232    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8f84c7-6449-4add-be75-f2e3819e04a7" containerName="payload-pre-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241240    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="af90c02b-8a15-411b-bbcb-d3d658effbec" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241249    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c30d58-399a-4f54-b1c5-96ed607bcf89" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241258    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4294be0-dd4b-4442-8f5e-9644df33121c" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241267    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0d20f7-3a1a-4875-84fb-29ae05de3304" containerName="main"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241277    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241286    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22001a2-054d-40d7-b536-4f1c26d5dbe2" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241296    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241304    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae638011-2505-459d-ad37-e3e618d710dc" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241313    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e5d59e-3940-42c0-906f-38bad87cb8ff" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241321    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbdd9f9d-6b83-495e-804f-e85d4185d396" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241331    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="50cb03b6-5f8c-4e62-9bb8-4285acb19238" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241339    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ddf97db-429a-47db-b503-56559682eb9c" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241349    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241359    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3153c1af-b6f5-482d-bcc6-c577c5ad697d" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241368    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87731e5-07ec-4842-8e0a-f98200c5acf3" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241377    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="656dbefb-91f4-48ce-afe3-742b52686788" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241385    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0f6281-84e4-4b7b-a7ec-2412bfb4da99" containerName="payload-processing"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241392    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef74e2dc-1fb5-414d-b969-5edc2fb704ef" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241397    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="de810876-cea8-4249-b707-7d20744b27ad" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241404    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="036bc02a-67d6-4d8e-86b6-44fba4ad6f1a" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241410    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="475d261a-7603-40c1-8e1a-6e62ee35a5b9" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241416    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae502d4-f440-470b-a8f9-957c672d6745" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241421    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1aa404d-1c9e-47ec-b67d-e3f8124a17eb" containerName="main"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241427    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="9645737c-61cc-475d-9832-d5dda1b7dee9" containerName="main"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241433    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="749359a0-c74c-4899-8505-89f60d2dc987" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241437    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c14236-fd6a-404b-8219-5ff6ba2f98f3" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241443    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="676be30a-0353-4bbf-8775-1219db088582" containerName="maas-api"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241448    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f45d142-8150-4eec-9dbb-58e94d3d171d" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241455    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="df70c75b-9dcd-44e9-8a6d-0195be290365" containerName="istio-proxy"
Oct 30 21:57:55.243449 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.241463    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8023d07-bf60-418f-b26c-1ef210b5f032" containerName="main"
Oct 30 21:57:55.247097 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.246528    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.254441 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.254412    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"]
Oct 30 21:57:55.337424 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.337333    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rfr\" (UniqueName: \"kubernetes.io/projected/4f0c9622-1d43-4bdc-9bab-54eba6149baa-kube-api-access-d6rfr\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.337598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.337440    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.337598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.337493    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.340537 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.340514    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:55.340654 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.340584    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:55.342484 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.342467    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:55.342598 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.342581    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:57:55.438128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.438092    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rfr\" (UniqueName: \"kubernetes.io/projected/4f0c9622-1d43-4bdc-9bab-54eba6149baa-kube-api-access-d6rfr\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.438348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.438151    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.438348 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.438207    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.439015 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.438994    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.440852 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.440827    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.449574 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.449548    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rfr\" (UniqueName: \"kubernetes.io/projected/4f0c9622-1d43-4bdc-9bab-54eba6149baa-kube-api-access-d6rfr\") pod \"maas-api-revoke-sub-c889178d-8wgnh\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.564533 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.564495    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:55.854809 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.854781    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"]
Oct 30 21:57:55.857187 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:57:55.857162    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0c9622_1d43_4bdc_9bab_54eba6149baa.slice/crio-a536538419f21e3e0de6db827b59e94e1d6a287d1707a4625a42ed2a76992495 WatchSource:0}: Error finding container a536538419f21e3e0de6db827b59e94e1d6a287d1707a4625a42ed2a76992495: Status 404 returned error can't find the container with id a536538419f21e3e0de6db827b59e94e1d6a287d1707a4625a42ed2a76992495
Oct 30 21:57:55.956512 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:55.956471    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh" event={"ID":"4f0c9622-1d43-4bdc-9bab-54eba6149baa","Type":"ContainerStarted","Data":"a536538419f21e3e0de6db827b59e94e1d6a287d1707a4625a42ed2a76992495"}
Oct 30 21:57:56.966422 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:56.966327    4633 generic.go:356] "Generic (PLEG): container finished" podID="4f0c9622-1d43-4bdc-9bab-54eba6149baa" containerID="fcbe2cbaee009f9bc64266e3fa1edb271c932be1b4b21ef77080bb56899595c7" exitCode=0
Oct 30 21:57:56.966850 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:56.966418    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh" event={"ID":"4f0c9622-1d43-4bdc-9bab-54eba6149baa","Type":"ContainerDied","Data":"fcbe2cbaee009f9bc64266e3fa1edb271c932be1b4b21ef77080bb56899595c7"}
Oct 30 21:57:58.192145 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.192109    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:58.272688 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.272592    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-openshift-ca\") pod \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") "
Oct 30 21:57:58.272857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.272714    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-xks-ca\") pod \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") "
Oct 30 21:57:58.272857 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.272736    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0c9622-1d43-4bdc-9bab-54eba6149baa-kube-api-access-d6rfr\" (UniqueName: \"kubernetes.io/projected/4f0c9622-1d43-4bdc-9bab-54eba6149baa-kube-api-access-d6rfr\") pod \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\" (UID: \"4f0c9622-1d43-4bdc-9bab-54eba6149baa\") "
Oct 30 21:57:58.273034 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.273007    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-openshift-ca" pod "4f0c9622-1d43-4bdc-9bab-54eba6149baa" (UID: "4f0c9622-1d43-4bdc-9bab-54eba6149baa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:58.275042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.275014    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-xks-ca" pod "4f0c9622-1d43-4bdc-9bab-54eba6149baa" (UID: "4f0c9622-1d43-4bdc-9bab-54eba6149baa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:58.275042 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.275037    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0c9622-1d43-4bdc-9bab-54eba6149baa-kube-api-access-d6rfr" pod "4f0c9622-1d43-4bdc-9bab-54eba6149baa" (UID: "4f0c9622-1d43-4bdc-9bab-54eba6149baa"). InnerVolumeSpecName "kube-api-access-d6rfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:58.374070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.374029    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-openshift-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:58.374070 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.374068    4633 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0c9622-1d43-4bdc-9bab-54eba6149baa-maas-api-xks-ca\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:58.374289 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.374083    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6rfr\" (UniqueName: \"kubernetes.io/projected/4f0c9622-1d43-4bdc-9bab-54eba6149baa-kube-api-access-d6rfr\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:58.994949 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.994921    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh"
Oct 30 21:57:58.995162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.994914    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c889178d-8wgnh" event={"ID":"4f0c9622-1d43-4bdc-9bab-54eba6149baa","Type":"ContainerDied","Data":"a536538419f21e3e0de6db827b59e94e1d6a287d1707a4625a42ed2a76992495"}
Oct 30 21:57:58.995162 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:58.995046    4633 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a536538419f21e3e0de6db827b59e94e1d6a287d1707a4625a42ed2a76992495"
Oct 30 21:57:59.557752 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:59.557698    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s" podUID="75efdde9-a98a-47e0-99cc-9f618bbe57e2" containerName="payload-processing" containerID="cri-o://dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0" gracePeriod=25
Oct 30 21:57:59.567272 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:57:59.567236    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq" podUID="6517c30e-d017-4879-a147-c9a10e29f3f0" containerName="payload-pre-processing" containerID="cri-o://448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0" gracePeriod=25
Oct 30 21:58:04.203988 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.203945    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main" containerID="cri-o://44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6" gracePeriod=45
Oct 30 21:58:04.463779 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.463712    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:58:04.539568 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.539539    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ce0d97-730b-4936-959b-795acbd830be-kube-api-access-pqkxg\" (UniqueName: \"kubernetes.io/projected/20ce0d97-730b-4936-959b-795acbd830be-kube-api-access-pqkxg\") pod \"20ce0d97-730b-4936-959b-795acbd830be\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") "
Oct 30 21:58:04.539790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.539598    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ce0d97-730b-4936-959b-795acbd830be-tls-certs\" (UniqueName: \"kubernetes.io/secret/20ce0d97-730b-4936-959b-795acbd830be-tls-certs\") pod \"20ce0d97-730b-4936-959b-795acbd830be\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") "
Oct 30 21:58:04.539790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.539631    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-home\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-home\") pod \"20ce0d97-730b-4936-959b-795acbd830be\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") "
Oct 30 21:58:04.539790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.539664    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-model-cache\") pod \"20ce0d97-730b-4936-959b-795acbd830be\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") "
Oct 30 21:58:04.539790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.539695    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-dshm\") pod \"20ce0d97-730b-4936-959b-795acbd830be\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") "
Oct 30 21:58:04.539790 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.539747    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-tmp-dir\") pod \"20ce0d97-730b-4936-959b-795acbd830be\" (UID: \"20ce0d97-730b-4936-959b-795acbd830be\") "
Oct 30 21:58:04.540114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.539966    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-home" pod "20ce0d97-730b-4936-959b-795acbd830be" (UID: "20ce0d97-730b-4936-959b-795acbd830be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.540114 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.539994    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-model-cache" pod "20ce0d97-730b-4936-959b-795acbd830be" (UID: "20ce0d97-730b-4936-959b-795acbd830be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.540239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.540129    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-tmp-dir" pod "20ce0d97-730b-4936-959b-795acbd830be" (UID: "20ce0d97-730b-4936-959b-795acbd830be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.540239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.540139    4633 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-home\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.540239 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.540179    4633 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-model-cache\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.542101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.542072    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-dshm" pod "20ce0d97-730b-4936-959b-795acbd830be" (UID: "20ce0d97-730b-4936-959b-795acbd830be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.542194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.542133    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ce0d97-730b-4936-959b-795acbd830be-kube-api-access-pqkxg" pod "20ce0d97-730b-4936-959b-795acbd830be" (UID: "20ce0d97-730b-4936-959b-795acbd830be"). InnerVolumeSpecName "kube-api-access-pqkxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:04.542194 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.542154    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ce0d97-730b-4936-959b-795acbd830be-tls-certs" pod "20ce0d97-730b-4936-959b-795acbd830be" (UID: "20ce0d97-730b-4936-959b-795acbd830be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:04.641384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.641316    4633 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-tmp-dir\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.641384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.641347    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqkxg\" (UniqueName: \"kubernetes.io/projected/20ce0d97-730b-4936-959b-795acbd830be-kube-api-access-pqkxg\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.641384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.641357    4633 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20ce0d97-730b-4936-959b-795acbd830be-tls-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.641384 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.641365    4633 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20ce0d97-730b-4936-959b-795acbd830be-dshm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.837171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.837075    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s" podUID="75efdde9-a98a-47e0-99cc-9f618bbe57e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:04.837171 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:04.837149    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq" podUID="6517c30e-d017-4879-a147-c9a10e29f3f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:05.030798 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.030756    4633 generic.go:356] "Generic (PLEG): container finished" podID="20ce0d97-730b-4936-959b-795acbd830be" containerID="44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6" exitCode=0
Oct 30 21:58:05.030984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.030815    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"
Oct 30 21:58:05.030984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.030840    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" event={"ID":"20ce0d97-730b-4936-959b-795acbd830be","Type":"ContainerDied","Data":"44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6"}
Oct 30 21:58:05.030984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.030885    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm" event={"ID":"20ce0d97-730b-4936-959b-795acbd830be","Type":"ContainerDied","Data":"513e4bbdff03a37b221444868aa640d081221a634280a54fff44d5e31bbaf538"}
Oct 30 21:58:05.030984 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.030902    4633 scope.go:122] "RemoveContainer" containerID="44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6"
Oct 30 21:58:05.042492 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.042470    4633 scope.go:122] "RemoveContainer" containerID="44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6"
Oct 30 21:58:05.042884 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:58:05.042862    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6\": container with ID starting with 44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6 not found: ID does not exist" containerID="44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6"
Oct 30 21:58:05.042965 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.042897    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6"} err="failed to get container status \"44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6\": rpc error: code = NotFound desc = could not find container \"44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6\": container with ID starting with 44e05293694b709532c779bc3e03668bf1fdef40c629ab4a2b5e46ddd4d304a6 not found: ID does not exist"
Oct 30 21:58:05.054772 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.054745    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"]
Oct 30 21:58:05.060818 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.060794    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-2-w2-05a342-kserve-d77cc9cd9-2trmm"]
Oct 30 21:58:05.790571 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:05.790533    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ce0d97-730b-4936-959b-795acbd830be" path="/var/lib/kubelet/pods/20ce0d97-730b-4936-959b-795acbd830be/volumes"
Oct 30 21:58:07.381656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:07.381621    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:58:13.075351 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:13.075314    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qvg44"]
Oct 30 21:58:14.836842 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:14.836799    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s" podUID="75efdde9-a98a-47e0-99cc-9f618bbe57e2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:14.837270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:14.836850    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq" podUID="6517c30e-d017-4879-a147-c9a10e29f3f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:58:17.644822 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:17.644780    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"]
Oct 30 21:58:17.645270 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:17.645063    4633 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" containerID="cri-o://d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f" gracePeriod=30
Oct 30 21:58:18.108717 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:18.108681    4633 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:58:18.108913 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:18.108738    4633 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:58:18.747378 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:18.747330    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86dd4f6dc-ggqpz_ef9d6d07-61cb-41de-ba16-fdfe4799c877/authorino/0.log"
Oct 30 21:58:18.972271 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:18.972241    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cnlnx_8a968bc1-6e7b-4391-82bf-00d03fc511fb/manager/0.log"
Oct 30 21:58:19.181957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.181919    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gbnpd_3fd3f3dc-8243-47ce-9edd-26c8b7974922/manager/0.log"
Oct 30 21:58:19.832161 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.832135    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:58:19.857487 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.857462    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:58:19.890903 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.890868    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75efdde9-a98a-47e0-99cc-9f618bbe57e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75efdde9-a98a-47e0-99cc-9f618bbe57e2-plugins-config-volume\") pod \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\" (UID: \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\") "
Oct 30 21:58:19.891089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.890932    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75efdde9-a98a-47e0-99cc-9f618bbe57e2-kube-api-access-8bgwm\" (UniqueName: \"kubernetes.io/projected/75efdde9-a98a-47e0-99cc-9f618bbe57e2-kube-api-access-8bgwm\") pod \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\" (UID: \"75efdde9-a98a-47e0-99cc-9f618bbe57e2\") "
Oct 30 21:58:19.891089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.891010    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6517c30e-d017-4879-a147-c9a10e29f3f0-kube-api-access-69stn\" (UniqueName: \"kubernetes.io/projected/6517c30e-d017-4879-a147-c9a10e29f3f0-kube-api-access-69stn\") pod \"6517c30e-d017-4879-a147-c9a10e29f3f0\" (UID: \"6517c30e-d017-4879-a147-c9a10e29f3f0\") "
Oct 30 21:58:19.891089 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.891036    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6517c30e-d017-4879-a147-c9a10e29f3f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6517c30e-d017-4879-a147-c9a10e29f3f0-plugins-config-volume\") pod \"6517c30e-d017-4879-a147-c9a10e29f3f0\" (UID: \"6517c30e-d017-4879-a147-c9a10e29f3f0\") "
Oct 30 21:58:19.891320 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.891299    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75efdde9-a98a-47e0-99cc-9f618bbe57e2-plugins-config-volume" pod "75efdde9-a98a-47e0-99cc-9f618bbe57e2" (UID: "75efdde9-a98a-47e0-99cc-9f618bbe57e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:19.891557 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.891526    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6517c30e-d017-4879-a147-c9a10e29f3f0-plugins-config-volume" pod "6517c30e-d017-4879-a147-c9a10e29f3f0" (UID: "6517c30e-d017-4879-a147-c9a10e29f3f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:19.893344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.893321    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75efdde9-a98a-47e0-99cc-9f618bbe57e2-kube-api-access-8bgwm" pod "75efdde9-a98a-47e0-99cc-9f618bbe57e2" (UID: "75efdde9-a98a-47e0-99cc-9f618bbe57e2"). InnerVolumeSpecName "kube-api-access-8bgwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:19.893344 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.893332    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6517c30e-d017-4879-a147-c9a10e29f3f0-kube-api-access-69stn" pod "6517c30e-d017-4879-a147-c9a10e29f3f0" (UID: "6517c30e-d017-4879-a147-c9a10e29f3f0"). InnerVolumeSpecName "kube-api-access-69stn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:19.901887 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.901858    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qvg44_75561d01-e2cb-49bf-9bec-69f1b4441b5d/limitador/0.log"
Oct 30 21:58:19.992816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.992777    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75efdde9-a98a-47e0-99cc-9f618bbe57e2-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:19.992816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.992809    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bgwm\" (UniqueName: \"kubernetes.io/projected/75efdde9-a98a-47e0-99cc-9f618bbe57e2-kube-api-access-8bgwm\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:19.992816 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.992820    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69stn\" (UniqueName: \"kubernetes.io/projected/6517c30e-d017-4879-a147-c9a10e29f3f0-kube-api-access-69stn\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:19.993068 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:19.992830    4633 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6517c30e-d017-4879-a147-c9a10e29f3f0-plugins-config-volume\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:20.107855 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.107767    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cpxbl_eb1ebff5-79ec-4034-aee6-76e8bd0f4514/manager/0.log"
Oct 30 21:58:20.125215 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.125177    4633 generic.go:356] "Generic (PLEG): container finished" podID="6517c30e-d017-4879-a147-c9a10e29f3f0" containerID="448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0" exitCode=0
Oct 30 21:58:20.125414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.125236    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"
Oct 30 21:58:20.125414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.125270    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq" event={"ID":"6517c30e-d017-4879-a147-c9a10e29f3f0","Type":"ContainerDied","Data":"448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0"}
Oct 30 21:58:20.125414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.125337    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq" event={"ID":"6517c30e-d017-4879-a147-c9a10e29f3f0","Type":"ContainerDied","Data":"2faa0c24096ba508a06fc2ec27ea661f2d5c0aa66cbdd63d5111bff2ae7209e9"}
Oct 30 21:58:20.125414 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.125367    4633 scope.go:122] "RemoveContainer" containerID="448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0"
Oct 30 21:58:20.127180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.127085    4633 generic.go:356] "Generic (PLEG): container finished" podID="75efdde9-a98a-47e0-99cc-9f618bbe57e2" containerID="dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0" exitCode=0
Oct 30 21:58:20.127180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.127129    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"
Oct 30 21:58:20.127367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.127130    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s" event={"ID":"75efdde9-a98a-47e0-99cc-9f618bbe57e2","Type":"ContainerDied","Data":"dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0"}
Oct 30 21:58:20.127367 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.127236    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s" event={"ID":"75efdde9-a98a-47e0-99cc-9f618bbe57e2","Type":"ContainerDied","Data":"ac7c50422a87ec90445b522788cbea491830c30809bf22bf8843416c7d3fb878"}
Oct 30 21:58:20.136593 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.136500    4633 scope.go:122] "RemoveContainer" containerID="448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0"
Oct 30 21:58:20.136863 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:58:20.136826    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0\": container with ID starting with 448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0 not found: ID does not exist" containerID="448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0"
Oct 30 21:58:20.136990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.136870    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0"} err="failed to get container status \"448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0\": rpc error: code = NotFound desc = could not find container \"448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0\": container with ID starting with 448555bc9234f59640c3311e0d45e980252d408ebc50097ada4bd9a0ad0d77c0 not found: ID does not exist"
Oct 30 21:58:20.136990 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.136895    4633 scope.go:122] "RemoveContainer" containerID="dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0"
Oct 30 21:58:20.147179 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.147155    4633 scope.go:122] "RemoveContainer" containerID="dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0"
Oct 30 21:58:20.147645 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:58:20.147618    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0\": container with ID starting with dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0 not found: ID does not exist" containerID="dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0"
Oct 30 21:58:20.147792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.147653    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0"} err="failed to get container status \"dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0\": rpc error: code = NotFound desc = could not find container \"dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0\": container with ID starting with dcea7fcdbad1795d7887ea38072355cd07091141f3a3c2bf35ff24a2785f4ed0 not found: ID does not exist"
Oct 30 21:58:20.149820 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.149800    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"]
Oct 30 21:58:20.154361 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.154339    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-05a342-5d75c76b99-27zbq"]
Oct 30 21:58:20.165164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.165137    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"]
Oct 30 21:58:20.169931 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.169912    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-05a342-5d59c7cbf-8jq6s"]
Oct 30 21:58:20.872415 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:20.872335    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd78b7fdb-wlnpl_325f7a58-6334-4000-ace2-a01551c15805/manager/0.log"
Oct 30 21:58:21.260792 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:21.260763    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4skfk_ee562c52-75cd-44c7-80f6-d1ea3b46e873/manager/0.log"
Oct 30 21:58:21.790172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:21.790143    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6517c30e-d017-4879-a147-c9a10e29f3f0" path="/var/lib/kubelet/pods/6517c30e-d017-4879-a147-c9a10e29f3f0/volumes"
Oct 30 21:58:21.790503 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:21.790491    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75efdde9-a98a-47e0-99cc-9f618bbe57e2" path="/var/lib/kubelet/pods/75efdde9-a98a-47e0-99cc-9f618bbe57e2/volumes"
Oct 30 21:58:22.124426 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:22.124348    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8rbvc_ed98730c-b2a7-43a4-8416-b17455ae777f/server/0.log"
Oct 30 21:58:22.911559 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:22.911529    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:58:23.028164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028083    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-data\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028154    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istiod-ca-cert\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028188    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-envoy\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028232    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-socket\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028273    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-certs\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028359 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028335    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-token\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-token\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028396    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-kube-api-access-djmp5\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-kube-api-access-djmp5\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028408    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-data" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.028555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028446    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-podinfo\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028555 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028538    4633 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-credential-socket\") pod \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\" (UID: \"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a\") "
Oct 30 21:58:23.028814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028580    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-socket" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.028814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028609    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-certs" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.028814 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028657    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istiod-ca-cert" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:23.028954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028851    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-credential-socket" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.028954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028921    4633 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-data\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.028954 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028943    4633 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istiod-ca-cert\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.029072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028957    4633 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.029072 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.028971    4633 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-workload-certs\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.030735 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.030706    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-token" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:23.030875 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.030817    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-podinfo" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:58:23.031148 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.031121    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-envoy" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.031252 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.031144    4633 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-kube-api-access-djmp5" pod "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" (UID: "c70d2000-c8ba-42aa-b81b-efcf8db5fa2a"). InnerVolumeSpecName "kube-api-access-djmp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:23.129639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.129604    4633 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-envoy\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.129639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.129632    4633 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-token\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.129639 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.129643    4633 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djmp5\" (UniqueName: \"kubernetes.io/projected/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-kube-api-access-djmp5\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.130128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.129652    4633 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-istio-podinfo\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.130128 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.129661    4633 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a-credential-socket\") on node \"ip-10-0-140-12.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.148316 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.148288    4633 generic.go:356] "Generic (PLEG): container finished" podID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerID="d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f" exitCode=0
Oct 30 21:58:23.148468 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.148335    4633 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"
Oct 30 21:58:23.148468 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.148377    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" event={"ID":"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a","Type":"ContainerDied","Data":"d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f"}
Oct 30 21:58:23.148468 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.148455    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4" event={"ID":"c70d2000-c8ba-42aa-b81b-efcf8db5fa2a","Type":"ContainerDied","Data":"dcb84d326de756d8abe2641d3e3ba27384df2bf9dce6baca55b8d498cc4c6101"}
Oct 30 21:58:23.148590 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.148476    4633 scope.go:122] "RemoveContainer" containerID="d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f"
Oct 30 21:58:23.159664 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.159643    4633 scope.go:122] "RemoveContainer" containerID="d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f"
Oct 30 21:58:23.159967 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:58:23.159939    4633 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f\": container with ID starting with d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f not found: ID does not exist" containerID="d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f"
Oct 30 21:58:23.160048 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.159975    4633 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f"} err="failed to get container status \"d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f\": rpc error: code = NotFound desc = could not find container \"d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f\": container with ID starting with d00aec5327e7c25943567f88497d691072bfc2091c5ddaef5b4e6c6500fca91f not found: ID does not exist"
Oct 30 21:58:23.170548 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.170523    4633 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"]
Oct 30 21:58:23.174397 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.174375    4633 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-tmpd4"]
Oct 30 21:58:23.237927 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.237899    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86dd4f6dc-ggqpz_ef9d6d07-61cb-41de-ba16-fdfe4799c877/authorino/0.log"
Oct 30 21:58:23.791254 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:23.791214    4633 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" path="/var/lib/kubelet/pods/c70d2000-c8ba-42aa-b81b-efcf8db5fa2a/volumes"
Oct 30 21:58:25.014044 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:58:25.014006    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:58:33.592124 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:33.592024    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd78b7fdb-wlnpl_325f7a58-6334-4000-ace2-a01551c15805/manager/0.log"
Oct 30 21:58:33.959942 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:33.959909    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4skfk_ee562c52-75cd-44c7-80f6-d1ea3b46e873/manager/0.log"
Oct 30 21:58:34.847410 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:34.847378    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8rbvc_ed98730c-b2a7-43a4-8416-b17455ae777f/server/0.log"
Oct 30 21:58:37.524523 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:37.524488    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c889178d-8wgnh_4f0c9622-1d43-4bdc-9bab-54eba6149baa/revoke-keys/0.log"
Oct 30 21:58:39.582698 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:39.582658    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86dd4f6dc-ggqpz_ef9d6d07-61cb-41de-ba16-fdfe4799c877/authorino/0.log"
Oct 30 21:58:39.796692 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:39.796636    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cnlnx_8a968bc1-6e7b-4391-82bf-00d03fc511fb/manager/0.log"
Oct 30 21:58:39.991418 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:39.991389    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gbnpd_3fd3f3dc-8243-47ce-9edd-26c8b7974922/manager/0.log"
Oct 30 21:58:40.692160 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:40.692122    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qvg44_75561d01-e2cb-49bf-9bec-69f1b4441b5d/limitador/0.log"
Oct 30 21:58:40.871322 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:40.871288    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cpxbl_eb1ebff5-79ec-4034-aee6-76e8bd0f4514/manager/0.log"
Oct 30 21:58:41.728617 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:41.728582    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt_543e365f-965c-4ae4-bd76-69a422e1dd6f/istio-proxy/0.log"
Oct 30 21:58:41.914345 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:41.914312    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6pl4g_2a4dcdda-32c7-4337-9d26-7de4e0edd073/discovery/0.log"
Oct 30 21:58:43.025807 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:43.025774    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-8tv48_cae247f2-d22b-4593-9935-071384bb7be9/istio-proxy/0.log"
Oct 30 21:58:43.381032 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:43.380946    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-msql6_bfa29907-7f35-4ba2-8303-54079a98cb5c/istio-proxy/1.log"
Oct 30 21:58:44.352081 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:44.352049    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g_9ac64e06-5eef-43d8-94e5-64698a0d7e52/istio-proxy/0.log"
Oct 30 21:58:44.519578 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:44.519547    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-q8b56_f1f5c570-a948-4d1c-b92a-de57a980d11b/payload-pre-processing/0.log"
Oct 30 21:58:44.686836 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:44.686804    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-nb8q8_d3af9a3e-eac9-4673-b9da-fc0dc10bab32/payload-processing/0.log"
Oct 30 21:58:45.321402 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:45.321370    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-msql6_bfa29907-7f35-4ba2-8303-54079a98cb5c/istio-proxy/0.log"
Oct 30 21:58:46.020573 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:46.020542    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-cx6kv_d94d40fd-549b-42c9-82ec-bdacd455c4ab/main/0.log"
Oct 30 21:58:46.190028 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:46.189990    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qcjfz_eee866b3-a796-49fb-92dc-a4c27f0aa9a6/main/0.log"
Oct 30 21:58:46.550576 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:46.550542    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pxkz8_50f5a97c-2b36-48ea-ba42-cee113124243/main/0.log"
Oct 30 21:58:46.734051 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:58:46.734014    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-lj99j_e1d828e0-1eea-42a3-b932-223813212485/main/0.log"
Oct 30 21:58:55.013929 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:58:55.013888    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:59:02.203908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:02.203876    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lbvj4_c6b17092-1d1d-40e0-b85f-28f05ab9f5f6/global-pull-secret-syncer/0.log"
Oct 30 21:59:02.464634 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:02.464532    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r2kvg_93767113-7f48-4ed1-bb24-d6caaa025dfb/konnectivity-agent/0.log"
Oct 30 21:59:02.683377 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:02.683335    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-12.us-east-2.compute.internal_070c8221ace6d9936bb76d676bde1ea5/haproxy/0.log"
Oct 30 21:59:09.927957 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:09.927884    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86dd4f6dc-ggqpz_ef9d6d07-61cb-41de-ba16-fdfe4799c877/authorino/0.log"
Oct 30 21:59:10.634638 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:10.634606    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cnlnx_8a968bc1-6e7b-4391-82bf-00d03fc511fb/manager/0.log"
Oct 30 21:59:10.670914 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:10.670883    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gbnpd_3fd3f3dc-8243-47ce-9edd-26c8b7974922/manager/0.log"
Oct 30 21:59:11.336970 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:11.336938    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qvg44_75561d01-e2cb-49bf-9bec-69f1b4441b5d/limitador/0.log"
Oct 30 21:59:11.403420 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:11.403389    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cpxbl_eb1ebff5-79ec-4034-aee6-76e8bd0f4514/manager/0.log"
Oct 30 21:59:13.461929 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:13.461848    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vfdj_97ac0ce7-c6e3-445a-9492-4976098b86f9/node-exporter/0.log"
Oct 30 21:59:13.487490 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:13.487464    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vfdj_97ac0ce7-c6e3-445a-9492-4976098b86f9/kube-rbac-proxy/0.log"
Oct 30 21:59:13.513117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:13.513088    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vfdj_97ac0ce7-c6e3-445a-9492-4976098b86f9/init-textfile/0.log"
Oct 30 21:59:14.038893 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.038859    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7pp58_003c3612-162c-445f-a595-2c1bcb37f9b2/prometheus-operator/0.log"
Oct 30 21:59:14.065183 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.065149    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-7pp58_003c3612-162c-445f-a595-2c1bcb37f9b2/kube-rbac-proxy/0.log"
Oct 30 21:59:14.095524 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.095493    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-rl755_1d2ab1b7-4ef5-4d49-9aae-37aada030edb/prometheus-operator-admission-webhook/0.log"
Oct 30 21:59:14.217442 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217409    4633 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"]
Oct 30 21:59:14.217871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217854    4633 cpu_manager.go:414] "containerMap: removing container" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy"
Oct 30 21:59:14.217871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217870    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d132a9-f682-4d56-9de2-aaaa58c7eb35" containerName="istio-proxy"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217890    4633 cpu_manager.go:414] "containerMap: removing container" podUID="75efdde9-a98a-47e0-99cc-9f618bbe57e2" containerName="payload-processing"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217896    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75efdde9-a98a-47e0-99cc-9f618bbe57e2" containerName="payload-processing"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217902    4633 cpu_manager.go:414] "containerMap: removing container" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217907    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217914    4633 cpu_manager.go:414] "containerMap: removing container" podUID="6517c30e-d017-4879-a147-c9a10e29f3f0" containerName="payload-pre-processing"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217919    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6517c30e-d017-4879-a147-c9a10e29f3f0" containerName="payload-pre-processing"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217931    4633 cpu_manager.go:414] "containerMap: removing container" podUID="4f0c9622-1d43-4bdc-9bab-54eba6149baa" containerName="revoke-keys"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217935    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0c9622-1d43-4bdc-9bab-54eba6149baa" containerName="revoke-keys"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217956    4633 cpu_manager.go:414] "containerMap: removing container" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217962    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217968    4633 cpu_manager.go:414] "containerMap: removing container" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217972    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217979    4633 cpu_manager.go:414] "containerMap: removing container" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.217984    4633 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218053    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff05b49-76c1-45dc-9f4d-3d07699c12d7" containerName="istio-proxy"
Oct 30 21:59:14.218054 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218063    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="75efdde9-a98a-47e0-99cc-9f618bbe57e2" containerName="payload-processing"
Oct 30 21:59:14.218838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218073    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ef3b75-43f3-44ea-a266-7184d31c4331" containerName="istio-proxy"
Oct 30 21:59:14.218838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218082    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ce0d97-730b-4936-959b-795acbd830be" containerName="main"
Oct 30 21:59:14.218838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218089    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="6517c30e-d017-4879-a147-c9a10e29f3f0" containerName="payload-pre-processing"
Oct 30 21:59:14.218838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218098    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="26abf544-f44b-43da-9cbf-50968294c4dc" containerName="istio-proxy"
Oct 30 21:59:14.218838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218110    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0c9622-1d43-4bdc-9bab-54eba6149baa" containerName="revoke-keys"
Oct 30 21:59:14.218838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218122    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70d2000-c8ba-42aa-b81b-efcf8db5fa2a" containerName="istio-proxy"
Oct 30 21:59:14.218838 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.218127    4633 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9785510-14ac-4cb6-bde5-b15c4cf827d4" containerName="istio-proxy"
Oct 30 21:59:14.220725 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.220709    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.233138 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.233115    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"]
Oct 30 21:59:14.313828 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.313729    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzjln\"/\"openshift-service-ca.crt\""
Oct 30 21:59:14.314020 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.313852    4633 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzjln\"/\"kube-root-ca.crt\""
Oct 30 21:59:14.314430 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.314411    4633 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qzjln\"/\"default-dockercfg-cbx5l\""
Oct 30 21:59:14.330180 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.330157    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-lib-modules\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.330336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.330200    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-sys\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.330336 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.330218    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-podres\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.330446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.330357    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-proc\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.330446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.330391    4633 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnhdh\" (UniqueName: \"kubernetes.io/projected/1f69fb0b-88d8-4112-8c39-e4f9168b2596-kube-api-access-fnhdh\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.431785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.431749    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-sys\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.431785 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.431783    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-podres\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.432031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.431838    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-proc\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.432031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.431866    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnhdh\" (UniqueName: \"kubernetes.io/projected/1f69fb0b-88d8-4112-8c39-e4f9168b2596-kube-api-access-fnhdh\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.432031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.431900    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-sys\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.432031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.431953    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-podres\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.432031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.431959    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-proc\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.432031 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.431992    4633 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-lib-modules\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.432283 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.432125    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f69fb0b-88d8-4112-8c39-e4f9168b2596-lib-modules\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.440731 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.440708    4633 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnhdh\" (UniqueName: \"kubernetes.io/projected/1f69fb0b-88d8-4112-8c39-e4f9168b2596-kube-api-access-fnhdh\") pod \"perf-node-gather-daemonset-m65zz\" (UID: \"1f69fb0b-88d8-4112-8c39-e4f9168b2596\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.534854 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.534818    4633 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:14.818572 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:14.818543    4633 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"]
Oct 30 21:59:14.819770 ip-10-0-140-12 kubenswrapper[4633]: W1030 21:59:14.819744    4633 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1f69fb0b_88d8_4112_8c39_e4f9168b2596.slice/crio-dd3fd032fac81a7503e78d6826a49d862296d7ea773773efe61b6bc8efa0c697 WatchSource:0}: Error finding container dd3fd032fac81a7503e78d6826a49d862296d7ea773773efe61b6bc8efa0c697: Status 404 returned error can't find the container with id dd3fd032fac81a7503e78d6826a49d862296d7ea773773efe61b6bc8efa0c697
Oct 30 21:59:15.453908 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:15.453872    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz" event={"ID":"1f69fb0b-88d8-4112-8c39-e4f9168b2596","Type":"ContainerStarted","Data":"ad9812baf25dc7356037a5afbee3673fa35a305e518dc159c3af19c8acef425e"}
Oct 30 21:59:15.454117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:15.453932    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:15.454117 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:15.453948    4633 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz" event={"ID":"1f69fb0b-88d8-4112-8c39-e4f9168b2596","Type":"ContainerStarted","Data":"dd3fd032fac81a7503e78d6826a49d862296d7ea773773efe61b6bc8efa0c697"}
Oct 30 21:59:15.469985 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:15.469937    4633 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz" podStartSLOduration=1.469923389 podStartE2EDuration="1.469923389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:15.468338919 +0000 UTC m=+3463.818066317" watchObservedRunningTime="2026-10-30 21:59:15.469923389 +0000 UTC m=+3463.819650785"
Oct 30 21:59:17.394941 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:17.394906    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-575dc7b45-6zxcw_32d59fa0-5d3d-4af4-a277-89de2848ae47/console/0.log"
Oct 30 21:59:17.437002 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:17.436968    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-7cf5k_332e4903-5c8f-42c9-8de5-5a7fdfb81679/download-server/0.log"
Oct 30 21:59:17.767488 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:17.767437    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd78b7fdb-wlnpl_325f7a58-6334-4000-ace2-a01551c15805/manager/0.log"
Oct 30 21:59:17.769871 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:17.769854    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd78b7fdb-wlnpl_325f7a58-6334-4000-ace2-a01551c15805/manager/0.log"
Oct 30 21:59:17.859079 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:17.859047    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4skfk_ee562c52-75cd-44c7-80f6-d1ea3b46e873/manager/0.log"
Oct 30 21:59:17.859709 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:17.859689    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4skfk_ee562c52-75cd-44c7-80f6-d1ea3b46e873/manager/0.log"
Oct 30 21:59:19.296749 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:19.296711    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8rbvc_ed98730c-b2a7-43a4-8416-b17455ae777f/server/0.log"
Oct 30 21:59:19.297164 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:19.296822    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8rbvc_ed98730c-b2a7-43a4-8416-b17455ae777f/server/0.log"
Oct 30 21:59:20.434923 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:20.434897    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5l9cm_243d8065-2175-4a20-b229-ebaae8e987fb/dns/0.log"
Oct 30 21:59:20.466584 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:20.466562    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5l9cm_243d8065-2175-4a20-b229-ebaae8e987fb/kube-rbac-proxy/0.log"
Oct 30 21:59:20.682288 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:20.682249    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r7gpg_2492dc3b-9893-45f5-af15-314312a0a18f/dns-node-resolver/0.log"
Oct 30 21:59:21.470950 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:21.470924    4633 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-m65zz"
Oct 30 21:59:21.524704 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:21.524660    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57d7c98f69-s5ln4_e1bcdfbe-ab13-4330-8e43-79b22175e1e5/registry/0.log"
Oct 30 21:59:21.616015 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:21.615988    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wrffd_595d39f4-1364-4e9e-87ae-09120d783bdd/node-ca/0.log"
Oct 30 21:59:21.957971 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:21.957940    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd78b7fdb-wlnpl_325f7a58-6334-4000-ace2-a01551c15805/manager/0.log"
Oct 30 21:59:22.024659 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:22.024617    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4skfk_ee562c52-75cd-44c7-80f6-d1ea3b46e873/manager/0.log"
Oct 30 21:59:22.069268 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:22.069235    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd78b7fdb-wlnpl_325f7a58-6334-4000-ace2-a01551c15805/manager/0.log"
Oct 30 21:59:22.142172 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:22.142134    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4skfk_ee562c52-75cd-44c7-80f6-d1ea3b46e873/manager/0.log"
Oct 30 21:59:23.374944 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:23.374917    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt_543e365f-965c-4ae4-bd76-69a422e1dd6f/istio-proxy/0.log"
Oct 30 21:59:23.494323 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:23.494295    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8rbvc_ed98730c-b2a7-43a4-8416-b17455ae777f/server/0.log"
Oct 30 21:59:23.627075 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:23.626983    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8rbvc_ed98730c-b2a7-43a4-8416-b17455ae777f/server/0.log"
Oct 30 21:59:25.024091 ip-10-0-140-12 kubenswrapper[4633]: E1030 21:59:25.024050    4633 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7hnm" containerName="extractor"
Oct 30 21:59:25.056793 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:25.056765    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6pl4g_2a4dcdda-32c7-4337-9d26-7de4e0edd073/discovery/0.log"
Oct 30 21:59:26.837101 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:26.837071    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd78b7fdb-wlnpl_325f7a58-6334-4000-ace2-a01551c15805/manager/0.log"
Oct 30 21:59:26.911417 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:26.911378    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4skfk_ee562c52-75cd-44c7-80f6-d1ea3b46e873/manager/0.log"
Oct 30 21:59:27.007656 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:27.007621    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd78b7fdb-wlnpl_325f7a58-6334-4000-ace2-a01551c15805/manager/0.log"
Oct 30 21:59:27.087713 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:27.087615    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-4skfk_ee562c52-75cd-44c7-80f6-d1ea3b46e873/manager/0.log"
Oct 30 21:59:28.748446 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:28.748421    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8rbvc_ed98730c-b2a7-43a4-8416-b17455ae777f/server/0.log"
Oct 30 21:59:28.927411 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:28.927383    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8rbvc_ed98730c-b2a7-43a4-8416-b17455ae777f/server/0.log"
Oct 30 21:59:29.147182 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:29.147102    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt_543e365f-965c-4ae4-bd76-69a422e1dd6f/istio-proxy/0.log"
Oct 30 21:59:30.749602 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:30.749551    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6pl4g_2a4dcdda-32c7-4337-9d26-7de4e0edd073/discovery/0.log"
Oct 30 21:59:30.759217 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:30.759193    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-8tv48_cae247f2-d22b-4593-9935-071384bb7be9/istio-proxy/0.log"
Oct 30 21:59:30.826848 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:30.826816    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-msql6_bfa29907-7f35-4ba2-8303-54079a98cb5c/istio-proxy/0.log"
Oct 30 21:59:30.829898 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:30.829877    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-msql6_bfa29907-7f35-4ba2-8303-54079a98cb5c/istio-proxy/1.log"
Oct 30 21:59:30.978463 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:30.978432    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-86dbb48f59-xg64g_9ac64e06-5eef-43d8-94e5-64698a0d7e52/istio-proxy/0.log"
Oct 30 21:59:31.025439 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:31.025368    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-q8b56_f1f5c570-a948-4d1c-b92a-de57a980d11b/payload-pre-processing/0.log"
Oct 30 21:59:31.097017 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:31.096987    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-nb8q8_d3af9a3e-eac9-4673-b9da-fc0dc10bab32/payload-processing/0.log"
Oct 30 21:59:32.336001 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:32.335957    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt_543e365f-965c-4ae4-bd76-69a422e1dd6f/istio-proxy/0.log"
Oct 30 21:59:32.451173 ip-10-0-140-12 kubenswrapper[4633]: I1030 21:59:32.451141    4633 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57d8447fc-v2zmt_543e365f-965c-4ae4-bd76-69a422e1dd6f/istio-proxy/0.log"